Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 07:59:23.407294 398 test_util.cc:268] Using random seed: -581155929
I20240501 07:59:23.414368 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 07:59:23.414428 398 ts_itest-base.cc:113] --------------
I20240501 07:59:23.414445 398 ts_itest-base.cc:114] 6 tablet servers
I20240501 07:59:23.414466 398 ts_itest-base.cc:115] 3 replicas per TS
I20240501 07:59:23.414489 398 ts_itest-base.cc:116] --------------
2024-05-01T07:59:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T07:59:23Z Disabled control of system clock
I20240501 07:59:23.423460 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:46217
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:34447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:46217
--location_mapping_cmd=/tmp/dist-test-taskS8gUPX/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 07:59:23.529793 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:23.530112 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:23.530190 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:23.533401 410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:23.533492 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 07:59:23.533529 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:23.533562 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 07:59:23.533598 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 07:59:23.537446 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskS8gUPX/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:46217
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:46217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:23.538442 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:23.539590 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 07:59:23.548203 410 server_base.cc:1034] running on GCE node
W20240501 07:59:23.548310 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:23.548921 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:23.550269 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:23.550807 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:23.551765 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:23.552932 410 hybrid_clock.cc:648] HybridClock initialized: now 1714550363552918 us; error 28 us; skew 500 ppm
I20240501 07:59:23.559585 410 webserver.cc:460] Webserver started at http://127.0.99.190:43773/ using document root <none> and password file <none>
I20240501 07:59:23.560233 410 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:23.560302 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:23.560487 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:23.562301 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1b115d81237d477ebb7ce82f63d5eac2"
format_stamp: "Formatted at 2024-05-01 07:59:23 on dist-test-slave-wslg"
I20240501 07:59:23.562801 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1b115d81237d477ebb7ce82f63d5eac2"
format_stamp: "Formatted at 2024-05-01 07:59:23 on dist-test-slave-wslg"
I20240501 07:59:23.566342 410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:23.568701 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:23.569718 410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:23.569849 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1b115d81237d477ebb7ce82f63d5eac2"
format_stamp: "Formatted at 2024-05-01 07:59:23 on dist-test-slave-wslg"
I20240501 07:59:23.569941 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:23.597513 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:23.598263 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:23.598448 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:23.605530 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:46217
I20240501 07:59:23.605589 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:46217 every 8 connection(s)
I20240501 07:59:23.606549 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 07:59:23.609258 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:23.609304 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 410
I20240501 07:59:23.609428 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 07:59:23.615397 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2: Bootstrap starting.
I20240501 07:59:23.617769 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:23.618544 477 log.cc:824] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:23.620386 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2: No bootstrap required, opened a new log
I20240501 07:59:23.623620 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b115d81237d477ebb7ce82f63d5eac2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46217 } }
I20240501 07:59:23.623857 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:23.623978 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b115d81237d477ebb7ce82f63d5eac2, State: Initialized, Role: FOLLOWER
I20240501 07:59:23.624534 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b115d81237d477ebb7ce82f63d5eac2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46217 } }
I20240501 07:59:23.624668 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:59:23.624769 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:59:23.624928 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:23.625784 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b115d81237d477ebb7ce82f63d5eac2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46217 } }
I20240501 07:59:23.626116 477 leader_election.cc:302] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b115d81237d477ebb7ce82f63d5eac2; no voters:
I20240501 07:59:23.626367 477 leader_election.cc:288] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:59:23.626474 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:23.627074 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [term 1 LEADER]: Becoming Leader. State: Replica: 1b115d81237d477ebb7ce82f63d5eac2, State: Running, Role: LEADER
I20240501 07:59:23.627384 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b115d81237d477ebb7ce82f63d5eac2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46217 } }
I20240501 07:59:23.627430 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:59:23.628818 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b115d81237d477ebb7ce82f63d5eac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b115d81237d477ebb7ce82f63d5eac2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46217 } } }
I20240501 07:59:23.628916 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:23.629055 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b115d81237d477ebb7ce82f63d5eac2. Latest consensus state: current_term: 1 leader_uuid: "1b115d81237d477ebb7ce82f63d5eac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b115d81237d477ebb7ce82f63d5eac2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46217 } } }
I20240501 07:59:23.629143 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2 [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:23.629436 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:59:23.631971 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:59:23.636677 491 catalog_manager.cc:1347] Generated new cluster ID: 58e64f506e184f5e9e04f38121a23819
I20240501 07:59:23.636763 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:59:23.665839 491 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:59:23.666859 491 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:59:23.676759 491 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1b115d81237d477ebb7ce82f63d5eac2: Generated new TSK 0
I20240501 07:59:23.677701 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:59:23.692106 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46217
--builtin_ntp_servers=127.0.99.148:34447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 07:59:23.802470 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:23.802711 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:23.802788 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:23.805830 501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:23.805902 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:23.806056 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:59:23.809752 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46217
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:23.810720 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:23.811862 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:23.820415 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:23.820506 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:23.821585 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:23.822445 501 server_base.cc:1034] running on GCE node
I20240501 07:59:23.822829 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:23.823412 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:23.824579 501 hybrid_clock.cc:648] HybridClock initialized: now 1714550363824567 us; error 31 us; skew 500 ppm
I20240501 07:59:23.826571 501 webserver.cc:460] Webserver started at http://127.0.99.129:38693/ using document root <none> and password file <none>
I20240501 07:59:23.827131 501 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:23.827198 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:23.827396 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:23.829213 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b9644aeae9de4deba0c3a2b85df04747"
format_stamp: "Formatted at 2024-05-01 07:59:23 on dist-test-slave-wslg"
I20240501 07:59:23.829774 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b9644aeae9de4deba0c3a2b85df04747"
format_stamp: "Formatted at 2024-05-01 07:59:23 on dist-test-slave-wslg"
I20240501 07:59:23.833246 501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:23.835462 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:23.836484 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:23.836588 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b9644aeae9de4deba0c3a2b85df04747"
format_stamp: "Formatted at 2024-05-01 07:59:23 on dist-test-slave-wslg"
I20240501 07:59:23.836674 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:23.850097 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:23.850768 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:23.850946 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:23.851511 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:23.852484 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:23.852535 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:23.852593 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:23.852624 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:23.862061 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:34609
I20240501 07:59:23.862139 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:34609 every 8 connection(s)
I20240501 07:59:23.863085 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:59:23.868022 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 501
I20240501 07:59:23.868121 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 07:59:23.870364 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46217
--builtin_ntp_servers=127.0.99.148:34447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:23.872752 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46217
I20240501 07:59:23.872985 629 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:23.873557 629 heartbeater.cc:511] Master 127.0.99.190:46217 requested a full tablet report, sending...
I20240501 07:59:23.924914 441 ts_manager.cc:194] Registered new tserver with Master: b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609)
I20240501 07:59:23.926404 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:57547
W20240501 07:59:23.997345 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:23.997599 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:23.997643 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:24.001278 632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:24.001361 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:24.001436 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 07:59:24.005215 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46217
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:24.006253 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:24.007396 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:24.015345 639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:24.016458 632 server_base.cc:1034] running on GCE node
W20240501 07:59:24.017336 641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:24.018076 638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:24.018467 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:24.019016 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:24.020191 632 hybrid_clock.cc:648] HybridClock initialized: now 1714550364020168 us; error 35 us; skew 500 ppm
I20240501 07:59:24.022125 632 webserver.cc:460] Webserver started at http://127.0.99.130:42703/ using document root <none> and password file <none>
I20240501 07:59:24.022682 632 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:24.022745 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:24.022933 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:24.024694 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3ee011df676c43f7beaeace0f8c60725"
format_stamp: "Formatted at 2024-05-01 07:59:24 on dist-test-slave-wslg"
I20240501 07:59:24.025219 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3ee011df676c43f7beaeace0f8c60725"
format_stamp: "Formatted at 2024-05-01 07:59:24 on dist-test-slave-wslg"
I20240501 07:59:24.028836 632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20240501 07:59:24.031255 647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:24.032285 632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:24.032397 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3ee011df676c43f7beaeace0f8c60725"
format_stamp: "Formatted at 2024-05-01 07:59:24 on dist-test-slave-wslg"
I20240501 07:59:24.032488 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:24.051260 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:24.051949 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:24.052127 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:24.052688 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:24.053614 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:24.053665 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:24.053723 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:24.053754 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:24.063839 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:40765
I20240501 07:59:24.063881 760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:40765 every 8 connection(s)
I20240501 07:59:24.064822 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 07:59:24.067173 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 632
I20240501 07:59:24.067278 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 07:59:24.069758 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46217
--builtin_ntp_servers=127.0.99.148:34447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:24.074800 761 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46217
I20240501 07:59:24.075026 761 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:24.075542 761 heartbeater.cc:511] Master 127.0.99.190:46217 requested a full tablet report, sending...
I20240501 07:59:24.110965 441 ts_manager.cc:194] Registered new tserver with Master: 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765)
I20240501 07:59:24.111930 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:49301
W20240501 07:59:24.184655 764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:24.184933 764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:24.184980 764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:24.188176 764 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:24.188267 764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:24.188338 764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:59:24.192302 764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46217
--never_fsync=true
--heap_profile_path=/tmp/kudu.764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:24.193257 764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:24.194458 764 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:24.202674 770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:24.203423 771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:24.204536 773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:24.205642 764 server_base.cc:1034] running on GCE node
I20240501 07:59:24.206107 764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:24.206701 764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:24.207859 764 hybrid_clock.cc:648] HybridClock initialized: now 1714550364207841 us; error 33 us; skew 500 ppm
I20240501 07:59:24.209820 764 webserver.cc:460] Webserver started at http://127.0.99.131:33207/ using document root <none> and password file <none>
I20240501 07:59:24.210419 764 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:24.210477 764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:24.210654 764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:24.212385 764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8ee66e185465431389043d933cb51308"
format_stamp: "Formatted at 2024-05-01 07:59:24 on dist-test-slave-wslg"
I20240501 07:59:24.212973 764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8ee66e185465431389043d933cb51308"
format_stamp: "Formatted at 2024-05-01 07:59:24 on dist-test-slave-wslg"
I20240501 07:59:24.216418 764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:59:24.218689 779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:24.219789 764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:24.219890 764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8ee66e185465431389043d933cb51308"
format_stamp: "Formatted at 2024-05-01 07:59:24 on dist-test-slave-wslg"
I20240501 07:59:24.219986 764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:24.241853 764 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:24.242580 764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:24.242746 764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:24.243330 764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:24.244288 764 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:24.244336 764 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:24.244395 764 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:24.244426 764 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:24.254093 892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:37541 every 8 connection(s)
I20240501 07:59:24.254069 764 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:37541
I20240501 07:59:24.255190 764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:59:24.256412 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 764
I20240501 07:59:24.256515 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 07:59:24.258878 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46217
--builtin_ntp_servers=127.0.99.148:34447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:24.265327 893 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46217
I20240501 07:59:24.265565 893 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:24.266172 893 heartbeater.cc:511] Master 127.0.99.190:46217 requested a full tablet report, sending...
I20240501 07:59:24.300799 441 ts_manager.cc:194] Registered new tserver with Master: 8ee66e185465431389043d933cb51308 (127.0.99.131:37541)
I20240501 07:59:24.301554 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:57571
W20240501 07:59:24.370530 896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:24.370780 896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:24.370855 896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:24.373948 896 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:24.374045 896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:24.374137 896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:59:24.377764 896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46217
--never_fsync=true
--heap_profile_path=/tmp/kudu.896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:24.378669 896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:24.379823 896 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:24.387899 902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:24.388610 905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:24.389578 903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:24.390722 896 server_base.cc:1034] running on GCE node
I20240501 07:59:24.391245 896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:24.391868 896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:24.393055 896 hybrid_clock.cc:648] HybridClock initialized: now 1714550364393045 us; error 29 us; skew 500 ppm
I20240501 07:59:24.395224 896 webserver.cc:460] Webserver started at http://127.0.99.132:34245/ using document root <none> and password file <none>
I20240501 07:59:24.395805 896 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:24.395864 896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:24.396051 896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:24.397749 896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1526c5008f714bcfa69e9bacb51a662c"
format_stamp: "Formatted at 2024-05-01 07:59:24 on dist-test-slave-wslg"
I20240501 07:59:24.398267 896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1526c5008f714bcfa69e9bacb51a662c"
format_stamp: "Formatted at 2024-05-01 07:59:24 on dist-test-slave-wslg"
I20240501 07:59:24.401665 896 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240501 07:59:24.403950 911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:24.405069 896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:24.405177 896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1526c5008f714bcfa69e9bacb51a662c"
format_stamp: "Formatted at 2024-05-01 07:59:24 on dist-test-slave-wslg"
I20240501 07:59:24.405272 896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:24.433266 896 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:24.433948 896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:24.434167 896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:24.434728 896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:24.435669 896 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:24.435719 896 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:24.435773 896 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:24.435801 896 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:24.445762 896 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:36663
I20240501 07:59:24.445804 1024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:36663 every 8 connection(s)
I20240501 07:59:24.446802 896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:59:24.455747 1025 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46217
I20240501 07:59:24.455930 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 896
I20240501 07:59:24.455992 1025 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:24.456066 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 07:59:24.456537 1025 heartbeater.cc:511] Master 127.0.99.190:46217 requested a full tablet report, sending...
I20240501 07:59:24.458271 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46217
--builtin_ntp_servers=127.0.99.148:34447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:24.498800 440 ts_manager.cc:194] Registered new tserver with Master: 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663)
I20240501 07:59:24.499562 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:50043
W20240501 07:59:24.576313 1029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:24.576545 1029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:24.576591 1029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:24.579756 1029 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:24.579843 1029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:24.579921 1029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:59:24.583863 1029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46217
--never_fsync=true
--heap_profile_path=/tmp/kudu.1029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:24.584942 1029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:24.586171 1029 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:24.594494 1034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:24.595301 1035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:24.596179 1037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:24.597002 1029 server_base.cc:1034] running on GCE node
I20240501 07:59:24.597401 1029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:24.598022 1029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:24.599181 1029 hybrid_clock.cc:648] HybridClock initialized: now 1714550364599167 us; error 32 us; skew 500 ppm
I20240501 07:59:24.601136 1029 webserver.cc:460] Webserver started at http://127.0.99.133:46751/ using document root <none> and password file <none>
I20240501 07:59:24.601698 1029 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:24.601760 1029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:24.601946 1029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:24.603703 1029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f5af98ef61b54ae0b33faeb78af96dec"
format_stamp: "Formatted at 2024-05-01 07:59:24 on dist-test-slave-wslg"
I20240501 07:59:24.604235 1029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f5af98ef61b54ae0b33faeb78af96dec"
format_stamp: "Formatted at 2024-05-01 07:59:24 on dist-test-slave-wslg"
I20240501 07:59:24.607786 1029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:59:24.610028 1043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:24.611233 1029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:24.611353 1029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f5af98ef61b54ae0b33faeb78af96dec"
format_stamp: "Formatted at 2024-05-01 07:59:24 on dist-test-slave-wslg"
I20240501 07:59:24.611445 1029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:24.631786 1029 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:24.632431 1029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:24.632598 1029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:24.633167 1029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:24.634171 1029 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:24.634236 1029 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:24.634291 1029 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:24.634323 1029 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:24.645004 1029 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:38665
I20240501 07:59:24.645025 1156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:38665 every 8 connection(s)
I20240501 07:59:24.646060 1029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:59:24.654709 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 1029
I20240501 07:59:24.654842 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 07:59:24.655205 1157 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46217
I20240501 07:59:24.655467 1157 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:24.656111 1157 heartbeater.cc:511] Master 127.0.99.190:46217 requested a full tablet report, sending...
I20240501 07:59:24.657323 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46217
--builtin_ntp_servers=127.0.99.148:34447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:24.694754 440 ts_manager.cc:194] Registered new tserver with Master: f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:24.695492 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:55549
W20240501 07:59:24.771750 1160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:24.771992 1160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:24.772061 1160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:24.775265 1160 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:24.775346 1160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:24.775453 1160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:59:24.779157 1160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46217
--never_fsync=true
--heap_profile_path=/tmp/kudu.1160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:24.780143 1160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:24.781328 1160 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:24.789781 1166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:24.790511 1167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:24.791445 1169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:24.792650 1160 server_base.cc:1034] running on GCE node
I20240501 07:59:24.793083 1160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:24.793687 1160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:24.794867 1160 hybrid_clock.cc:648] HybridClock initialized: now 1714550364794855 us; error 32 us; skew 500 ppm
I20240501 07:59:24.796782 1160 webserver.cc:460] Webserver started at http://127.0.99.134:35945/ using document root <none> and password file <none>
I20240501 07:59:24.797360 1160 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:24.797425 1160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:24.797608 1160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:24.799360 1160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "c45c306768564fb2bd741c535d334429"
format_stamp: "Formatted at 2024-05-01 07:59:24 on dist-test-slave-wslg"
I20240501 07:59:24.799844 1160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "c45c306768564fb2bd741c535d334429"
format_stamp: "Formatted at 2024-05-01 07:59:24 on dist-test-slave-wslg"
I20240501 07:59:24.803213 1160 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240501 07:59:24.805553 1175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:24.806798 1160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:24.806896 1160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c45c306768564fb2bd741c535d334429"
format_stamp: "Formatted at 2024-05-01 07:59:24 on dist-test-slave-wslg"
I20240501 07:59:24.806998 1160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:24.832727 1160 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:24.833462 1160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:24.833627 1160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:24.834259 1160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:24.835175 1160 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:24.835222 1160 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:24.835260 1160 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:24.835290 1160 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:24.846294 1160 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:37067
I20240501 07:59:24.846371 1288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:37067 every 8 connection(s)
I20240501 07:59:24.847303 1160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:59:24.853838 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 1160
I20240501 07:59:24.853955 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 07:59:24.857023 1289 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46217
I20240501 07:59:24.857275 1289 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:24.857851 1289 heartbeater.cc:511] Master 127.0.99.190:46217 requested a full tablet report, sending...
I20240501 07:59:24.893060 440 ts_manager.cc:194] Registered new tserver with Master: c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:24.893779 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:51519
I20240501 07:59:24.894135 398 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 07:59:24.906152 398 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 07:59:24.929075 629 heartbeater.cc:503] Master 127.0.99.190:46217 was elected leader, sending a full tablet report...
I20240501 07:59:25.114425 761 heartbeater.cc:503] Master 127.0.99.190:46217 was elected leader, sending a full tablet report...
I20240501 07:59:25.303896 893 heartbeater.cc:503] Master 127.0.99.190:46217 was elected leader, sending a full tablet report...
I20240501 07:59:25.501827 1025 heartbeater.cc:503] Master 127.0.99.190:46217 was elected leader, sending a full tablet report...
I20240501 07:59:25.697664 1157 heartbeater.cc:503] Master 127.0.99.190:46217 was elected leader, sending a full tablet report...
I20240501 07:59:25.896322 1289 heartbeater.cc:503] Master 127.0.99.190:46217 was elected leader, sending a full tablet report...
I20240501 07:59:28.658792 440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36176:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:28.681542 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 0511b10a941e4dd084a0bc667cf5298d (DEFAULT_TABLE table=rebalance_test_table_0 [id=62e911d4c4af4569bbd20c478accea72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:28.682704 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0511b10a941e4dd084a0bc667cf5298d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:28.682420 563 tablet_service.cc:1467] Processing CreateTablet for tablet 77f7440c254d4232b19565899d33f4a2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=62e911d4c4af4569bbd20c478accea72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:28.683279 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77f7440c254d4232b19565899d33f4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:28.683677 827 tablet_service.cc:1467] Processing CreateTablet for tablet 77f7440c254d4232b19565899d33f4a2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=62e911d4c4af4569bbd20c478accea72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:28.684352 562 tablet_service.cc:1467] Processing CreateTablet for tablet 0511b10a941e4dd084a0bc667cf5298d (DEFAULT_TABLE table=rebalance_test_table_0 [id=62e911d4c4af4569bbd20c478accea72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:28.684515 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77f7440c254d4232b19565899d33f4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:28.684599 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0511b10a941e4dd084a0bc667cf5298d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:28.686285 826 tablet_service.cc:1467] Processing CreateTablet for tablet 0511b10a941e4dd084a0bc667cf5298d (DEFAULT_TABLE table=rebalance_test_table_0 [id=62e911d4c4af4569bbd20c478accea72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:28.686559 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0511b10a941e4dd084a0bc667cf5298d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:28.687958 959 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9333d0caf54332b6a6dc3fe643c35a (DEFAULT_TABLE table=rebalance_test_table_0 [id=62e911d4c4af4569bbd20c478accea72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:28.688832 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9333d0caf54332b6a6dc3fe643c35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:28.690428 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9333d0caf54332b6a6dc3fe643c35a (DEFAULT_TABLE table=rebalance_test_table_0 [id=62e911d4c4af4569bbd20c478accea72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:28.690721 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9333d0caf54332b6a6dc3fe643c35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:28.692242 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 77f7440c254d4232b19565899d33f4a2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=62e911d4c4af4569bbd20c478accea72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:28.693022 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77f7440c254d4232b19565899d33f4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:28.695340 1315 tablet_bootstrap.cc:492] T 8f9333d0caf54332b6a6dc3fe643c35a P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:28.697012 1316 tablet_bootstrap.cc:492] T 8f9333d0caf54332b6a6dc3fe643c35a P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:28.697811 1315 tablet_bootstrap.cc:654] T 8f9333d0caf54332b6a6dc3fe643c35a P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:28.698699 1315 log.cc:824] T 8f9333d0caf54332b6a6dc3fe643c35a P 1526c5008f714bcfa69e9bacb51a662c: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:28.699236 1316 tablet_bootstrap.cc:654] T 8f9333d0caf54332b6a6dc3fe643c35a P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:28.699501 1314 tablet_bootstrap.cc:492] T 77f7440c254d4232b19565899d33f4a2 P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:28.700253 1316 log.cc:824] T 8f9333d0caf54332b6a6dc3fe643c35a P f5af98ef61b54ae0b33faeb78af96dec: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:28.699919 695 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9333d0caf54332b6a6dc3fe643c35a (DEFAULT_TABLE table=rebalance_test_table_0 [id=62e911d4c4af4569bbd20c478accea72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:28.700901 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9333d0caf54332b6a6dc3fe643c35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:28.701581 1315 tablet_bootstrap.cc:492] T 8f9333d0caf54332b6a6dc3fe643c35a P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:28.701771 1315 ts_tablet_manager.cc:1367] T 8f9333d0caf54332b6a6dc3fe643c35a P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 07:59:28.701916 1316 tablet_bootstrap.cc:492] T 8f9333d0caf54332b6a6dc3fe643c35a P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:28.702111 1316 ts_tablet_manager.cc:1367] T 8f9333d0caf54332b6a6dc3fe643c35a P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20240501 07:59:28.705901 1315 raft_consensus.cc:348] T 8f9333d0caf54332b6a6dc3fe643c35a P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.706193 1315 raft_consensus.cc:374] T 8f9333d0caf54332b6a6dc3fe643c35a P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:28.706249 1315 raft_consensus.cc:729] T 8f9333d0caf54332b6a6dc3fe643c35a P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:28.706141 1316 raft_consensus.cc:348] T 8f9333d0caf54332b6a6dc3fe643c35a P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.706321 1316 raft_consensus.cc:374] T 8f9333d0caf54332b6a6dc3fe643c35a P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:28.706365 1316 raft_consensus.cc:729] T 8f9333d0caf54332b6a6dc3fe643c35a P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:28.706804 1316 consensus_queue.cc:260] T 8f9333d0caf54332b6a6dc3fe643c35a P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.708194 1316 ts_tablet_manager.cc:1398] T 8f9333d0caf54332b6a6dc3fe643c35a P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 07:59:28.708591 1316 tablet_bootstrap.cc:492] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:28.709949 1316 tablet_bootstrap.cc:654] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:28.711437 1316 tablet_bootstrap.cc:492] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:28.711524 1316 ts_tablet_manager.cc:1367] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:59:28.712244 1316 raft_consensus.cc:348] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.712368 1316 raft_consensus.cc:374] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:28.712457 1316 raft_consensus.cc:729] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:28.712620 1316 consensus_queue.cc:260] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.712916 1316 ts_tablet_manager.cc:1398] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:28.713440 1315 consensus_queue.cc:260] T 8f9333d0caf54332b6a6dc3fe643c35a P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.714370 1315 ts_tablet_manager.cc:1398] T 8f9333d0caf54332b6a6dc3fe643c35a P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20240501 07:59:28.715008 1319 tablet_bootstrap.cc:492] T 0511b10a941e4dd084a0bc667cf5298d P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:28.715291 1324 tablet_bootstrap.cc:492] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:28.715888 1314 tablet_bootstrap.cc:654] T 77f7440c254d4232b19565899d33f4a2 P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:28.716876 1314 log.cc:824] T 77f7440c254d4232b19565899d33f4a2 P 8ee66e185465431389043d933cb51308: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:28.717425 1319 tablet_bootstrap.cc:654] T 0511b10a941e4dd084a0bc667cf5298d P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:28.717428 1324 tablet_bootstrap.cc:654] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:28.718420 1319 log.cc:824] T 0511b10a941e4dd084a0bc667cf5298d P b9644aeae9de4deba0c3a2b85df04747: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:28.718681 1324 log.cc:824] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:28.718819 1314 tablet_bootstrap.cc:492] T 77f7440c254d4232b19565899d33f4a2 P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:28.719048 1314 ts_tablet_manager.cc:1367] T 77f7440c254d4232b19565899d33f4a2 P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20240501 07:59:28.719336 1322 tablet_bootstrap.cc:492] T 77f7440c254d4232b19565899d33f4a2 P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:28.720118 1319 tablet_bootstrap.cc:492] T 0511b10a941e4dd084a0bc667cf5298d P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:28.720271 1319 ts_tablet_manager.cc:1367] T 0511b10a941e4dd084a0bc667cf5298d P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:59:28.721544 1322 tablet_bootstrap.cc:654] T 77f7440c254d4232b19565899d33f4a2 P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:28.722370 1322 log.cc:824] T 77f7440c254d4232b19565899d33f4a2 P c45c306768564fb2bd741c535d334429: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:28.723157 1314 raft_consensus.cc:348] T 77f7440c254d4232b19565899d33f4a2 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:28.723373 1314 raft_consensus.cc:374] T 77f7440c254d4232b19565899d33f4a2 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:28.723434 1314 raft_consensus.cc:729] T 77f7440c254d4232b19565899d33f4a2 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:28.723897 1314 consensus_queue.cc:260] T 77f7440c254d4232b19565899d33f4a2 P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:28.724013 1324 tablet_bootstrap.cc:492] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:28.724172 1324 ts_tablet_manager.cc:1367] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20240501 07:59:28.724079 1319 raft_consensus.cc:348] T 0511b10a941e4dd084a0bc667cf5298d P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.724265 1319 raft_consensus.cc:374] T 0511b10a941e4dd084a0bc667cf5298d P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:28.724316 1319 raft_consensus.cc:729] T 0511b10a941e4dd084a0bc667cf5298d P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:28.724687 1319 consensus_queue.cc:260] T 0511b10a941e4dd084a0bc667cf5298d P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.724843 1314 ts_tablet_manager.cc:1398] T 77f7440c254d4232b19565899d33f4a2 P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 07:59:28.725553 1319 ts_tablet_manager.cc:1398] T 0511b10a941e4dd084a0bc667cf5298d P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 07:59:28.725806 1319 tablet_bootstrap.cc:492] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:28.726895 1319 tablet_bootstrap.cc:654] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:28.726897 1314 tablet_bootstrap.cc:492] T 0511b10a941e4dd084a0bc667cf5298d P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:28.728065 1314 tablet_bootstrap.cc:654] T 0511b10a941e4dd084a0bc667cf5298d P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:28.728076 1324 raft_consensus.cc:348] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.728343 1324 raft_consensus.cc:374] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:28.728407 1324 raft_consensus.cc:729] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:28.728915 1324 consensus_queue.cc:260] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.729234 1322 tablet_bootstrap.cc:492] T 77f7440c254d4232b19565899d33f4a2 P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:28.729483 1322 ts_tablet_manager.cc:1367] T 77f7440c254d4232b19565899d33f4a2 P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240501 07:59:28.729854 1324 ts_tablet_manager.cc:1398] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 07:59:28.731207 1319 tablet_bootstrap.cc:492] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:28.731297 1319 ts_tablet_manager.cc:1367] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:28.731947 1319 raft_consensus.cc:348] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:28.732071 1319 raft_consensus.cc:374] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:28.732106 1319 raft_consensus.cc:729] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:28.732239 1319 consensus_queue.cc:260] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:28.732565 1319 ts_tablet_manager.cc:1398] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:28.733496 1314 tablet_bootstrap.cc:492] T 0511b10a941e4dd084a0bc667cf5298d P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:28.733636 1314 ts_tablet_manager.cc:1367] T 0511b10a941e4dd084a0bc667cf5298d P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 07:59:28.733564 1322 raft_consensus.cc:348] T 77f7440c254d4232b19565899d33f4a2 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:28.733805 1322 raft_consensus.cc:374] T 77f7440c254d4232b19565899d33f4a2 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:28.733872 1322 raft_consensus.cc:729] T 77f7440c254d4232b19565899d33f4a2 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:28.734355 1314 raft_consensus.cc:348] T 0511b10a941e4dd084a0bc667cf5298d P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.734354 1322 consensus_queue.cc:260] T 77f7440c254d4232b19565899d33f4a2 P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:28.734486 1314 raft_consensus.cc:374] T 0511b10a941e4dd084a0bc667cf5298d P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:28.734524 1314 raft_consensus.cc:729] T 0511b10a941e4dd084a0bc667cf5298d P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:28.734679 1314 consensus_queue.cc:260] T 0511b10a941e4dd084a0bc667cf5298d P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.735031 1314 ts_tablet_manager.cc:1398] T 0511b10a941e4dd084a0bc667cf5298d P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:28.735266 1322 ts_tablet_manager.cc:1398] T 77f7440c254d4232b19565899d33f4a2 P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20240501 07:59:28.759663 894 tablet.cc:2369] T 0511b10a941e4dd084a0bc667cf5298d P 8ee66e185465431389043d933cb51308: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:59:28.819501 762 tablet.cc:2369] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:28.850282 1330 raft_consensus.cc:482] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:28.850529 1330 raft_consensus.cc:504] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
W20240501 07:59:28.851606 1290 tablet.cc:2369] T 77f7440c254d4232b19565899d33f4a2 P c45c306768564fb2bd741c535d334429: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:28.851796 1330 leader_election.cc:288] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c45c306768564fb2bd741c535d334429 (127.0.99.134:37067), 8ee66e185465431389043d933cb51308 (127.0.99.131:37541)
I20240501 07:59:28.852867 1320 raft_consensus.cc:482] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:28.853099 1320 raft_consensus.cc:504] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.854527 1320 leader_election.cc:288] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609), 8ee66e185465431389043d933cb51308 (127.0.99.131:37541)
I20240501 07:59:28.857096 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77f7440c254d4232b19565899d33f4a2" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429" is_pre_election: true
I20240501 07:59:28.857169 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77f7440c254d4232b19565899d33f4a2" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308" is_pre_election: true
I20240501 07:59:28.857416 1243 raft_consensus.cc:2454] T 77f7440c254d4232b19565899d33f4a2 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 0.
I20240501 07:59:28.857460 847 raft_consensus.cc:2454] T 77f7440c254d4232b19565899d33f4a2 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 0.
I20240501 07:59:28.858086 516 leader_election.cc:302] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9644aeae9de4deba0c3a2b85df04747, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:28.858378 1330 raft_consensus.cc:2781] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:28.858448 1330 raft_consensus.cc:482] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:28.858489 1330 raft_consensus.cc:3037] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:28.859565 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0511b10a941e4dd084a0bc667cf5298d" candidate_uuid: "f5af98ef61b54ae0b33faeb78af96dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308" is_pre_election: true
I20240501 07:59:28.859637 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0511b10a941e4dd084a0bc667cf5298d" candidate_uuid: "f5af98ef61b54ae0b33faeb78af96dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747" is_pre_election: true
I20240501 07:59:28.859706 847 raft_consensus.cc:2454] T 0511b10a941e4dd084a0bc667cf5298d P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5af98ef61b54ae0b33faeb78af96dec in term 0.
I20240501 07:59:28.859802 583 raft_consensus.cc:2454] T 0511b10a941e4dd084a0bc667cf5298d P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5af98ef61b54ae0b33faeb78af96dec in term 0.
I20240501 07:59:28.860076 1046 leader_election.cc:302] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:28.860343 1320 raft_consensus.cc:2781] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:28.860409 1320 raft_consensus.cc:482] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:28.860438 1320 raft_consensus.cc:3037] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
W20240501 07:59:28.867628 630 tablet.cc:2369] T 0511b10a941e4dd084a0bc667cf5298d P b9644aeae9de4deba0c3a2b85df04747: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:59:28.900311 1158 tablet.cc:2369] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:28.902849 1332 raft_consensus.cc:482] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:28.903038 1332 raft_consensus.cc:504] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.904140 1332 leader_election.cc:288] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:28.908761 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9333d0caf54332b6a6dc3fe643c35a" candidate_uuid: "3ee011df676c43f7beaeace0f8c60725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec" is_pre_election: true
I20240501 07:59:28.908831 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9333d0caf54332b6a6dc3fe643c35a" candidate_uuid: "3ee011df676c43f7beaeace0f8c60725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1526c5008f714bcfa69e9bacb51a662c" is_pre_election: true
I20240501 07:59:28.908901 1330 raft_consensus.cc:504] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:28.909040 1106 raft_consensus.cc:2454] T 8f9333d0caf54332b6a6dc3fe643c35a P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ee011df676c43f7beaeace0f8c60725 in term 0.
I20240501 07:59:28.909096 979 raft_consensus.cc:2454] T 8f9333d0caf54332b6a6dc3fe643c35a P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ee011df676c43f7beaeace0f8c60725 in term 0.
I20240501 07:59:28.909456 1330 leader_election.cc:288] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 election: Requested vote from peers c45c306768564fb2bd741c535d334429 (127.0.99.134:37067), 8ee66e185465431389043d933cb51308 (127.0.99.131:37541)
I20240501 07:59:28.909482 1320 raft_consensus.cc:504] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.909621 649 leader_election.cc:302] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, 3ee011df676c43f7beaeace0f8c60725; no voters:
I20240501 07:59:28.909852 1320 leader_election.cc:288] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [CANDIDATE]: Term 1 election: Requested vote from peers b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609), 8ee66e185465431389043d933cb51308 (127.0.99.131:37541)
I20240501 07:59:28.909929 1332 raft_consensus.cc:2781] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:28.910018 1332 raft_consensus.cc:482] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:28.910076 1332 raft_consensus.cc:3037] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:28.910281 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77f7440c254d4232b19565899d33f4a2" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429"
I20240501 07:59:28.910420 1243 raft_consensus.cc:3037] T 77f7440c254d4232b19565899d33f4a2 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:28.910638 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0511b10a941e4dd084a0bc667cf5298d" candidate_uuid: "f5af98ef61b54ae0b33faeb78af96dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747"
I20240501 07:59:28.910753 583 raft_consensus.cc:3037] T 0511b10a941e4dd084a0bc667cf5298d P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:28.911237 1332 raft_consensus.cc:504] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.911597 1332 leader_election.cc:288] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [CANDIDATE]: Term 1 election: Requested vote from peers 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:28.911732 1243 raft_consensus.cc:2454] T 77f7440c254d4232b19565899d33f4a2 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 1.
I20240501 07:59:28.911792 583 raft_consensus.cc:2454] T 0511b10a941e4dd084a0bc667cf5298d P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5af98ef61b54ae0b33faeb78af96dec in term 1.
I20240501 07:59:28.911872 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77f7440c254d4232b19565899d33f4a2" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308"
I20240501 07:59:28.912003 847 raft_consensus.cc:3037] T 77f7440c254d4232b19565899d33f4a2 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:28.912151 1045 leader_election.cc:302] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9644aeae9de4deba0c3a2b85df04747, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:28.912336 1320 raft_consensus.cc:2781] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:28.912571 1320 raft_consensus.cc:686] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [term 1 LEADER]: Becoming Leader. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Running, Role: LEADER
I20240501 07:59:28.912663 516 leader_election.cc:302] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9644aeae9de4deba0c3a2b85df04747, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:28.912928 1320 consensus_queue.cc:237] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:28.913120 847 raft_consensus.cc:2454] T 77f7440c254d4232b19565899d33f4a2 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 1.
I20240501 07:59:28.913246 1330 raft_consensus.cc:2781] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:28.913475 1330 raft_consensus.cc:686] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [term 1 LEADER]: Becoming Leader. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Running, Role: LEADER
I20240501 07:59:28.913794 1330 consensus_queue.cc:237] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:28.914127 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9333d0caf54332b6a6dc3fe643c35a" candidate_uuid: "3ee011df676c43f7beaeace0f8c60725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1526c5008f714bcfa69e9bacb51a662c"
I20240501 07:59:28.914252 979 raft_consensus.cc:3037] T 8f9333d0caf54332b6a6dc3fe643c35a P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:28.916049 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0511b10a941e4dd084a0bc667cf5298d" candidate_uuid: "f5af98ef61b54ae0b33faeb78af96dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308"
I20240501 07:59:28.916188 847 raft_consensus.cc:3037] T 0511b10a941e4dd084a0bc667cf5298d P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:28.916935 441 catalog_manager.cc:5484] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b9644aeae9de4deba0c3a2b85df04747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:28.917222 439 catalog_manager.cc:5484] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec reported cstate change: term changed from 0 to 1, leader changed from <none> to f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133). New cstate: current_term: 1 leader_uuid: "f5af98ef61b54ae0b33faeb78af96dec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:28.917639 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9333d0caf54332b6a6dc3fe643c35a" candidate_uuid: "3ee011df676c43f7beaeace0f8c60725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec"
I20240501 07:59:28.917747 1106 raft_consensus.cc:3037] T 8f9333d0caf54332b6a6dc3fe643c35a P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
W20240501 07:59:28.952005 1026 tablet.cc:2369] T 8f9333d0caf54332b6a6dc3fe643c35a P 1526c5008f714bcfa69e9bacb51a662c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:29.008455 1106 raft_consensus.cc:2454] T 8f9333d0caf54332b6a6dc3fe643c35a P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ee011df676c43f7beaeace0f8c60725 in term 1.
I20240501 07:59:29.008455 847 raft_consensus.cc:2454] T 0511b10a941e4dd084a0bc667cf5298d P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5af98ef61b54ae0b33faeb78af96dec in term 1.
I20240501 07:59:29.008455 979 raft_consensus.cc:2454] T 8f9333d0caf54332b6a6dc3fe643c35a P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ee011df676c43f7beaeace0f8c60725 in term 1.
I20240501 07:59:29.009011 649 leader_election.cc:302] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, 3ee011df676c43f7beaeace0f8c60725, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:29.009287 1332 raft_consensus.cc:2781] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:29.009577 1332 raft_consensus.cc:686] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [term 1 LEADER]: Becoming Leader. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Running, Role: LEADER
I20240501 07:59:29.010033 1332 consensus_queue.cc:237] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:29.013970 439 catalog_manager.cc:5484] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3ee011df676c43f7beaeace0f8c60725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:29.168990 1348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:29.169339 1348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:29.172781 1348 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 07:59:29.285176 1106 raft_consensus.cc:1261] T 8f9333d0caf54332b6a6dc3fe643c35a P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Refusing update from remote peer 3ee011df676c43f7beaeace0f8c60725: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:29.285429 979 raft_consensus.cc:1261] T 8f9333d0caf54332b6a6dc3fe643c35a P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Refusing update from remote peer 3ee011df676c43f7beaeace0f8c60725: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:29.287223 847 raft_consensus.cc:1261] T 0511b10a941e4dd084a0bc667cf5298d P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Refusing update from remote peer f5af98ef61b54ae0b33faeb78af96dec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:29.287639 1243 raft_consensus.cc:1261] T 77f7440c254d4232b19565899d33f4a2 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Refusing update from remote peer b9644aeae9de4deba0c3a2b85df04747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:29.287230 846 raft_consensus.cc:1261] T 77f7440c254d4232b19565899d33f4a2 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Refusing update from remote peer b9644aeae9de4deba0c3a2b85df04747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:29.287855 1342 consensus_queue.cc:1035] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:29.288424 1330 consensus_queue.cc:1035] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:29.288754 583 raft_consensus.cc:1261] T 0511b10a941e4dd084a0bc667cf5298d P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Refusing update from remote peer f5af98ef61b54ae0b33faeb78af96dec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:29.289122 1332 consensus_queue.cc:1035] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:29.289747 1330 consensus_queue.cc:1035] T 77f7440c254d4232b19565899d33f4a2 P b9644aeae9de4deba0c3a2b85df04747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:29.293071 1342 consensus_queue.cc:1035] T 0511b10a941e4dd084a0bc667cf5298d P f5af98ef61b54ae0b33faeb78af96dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:29.293099 1332 consensus_queue.cc:1035] T 8f9333d0caf54332b6a6dc3fe643c35a P 3ee011df676c43f7beaeace0f8c60725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:29.303228 1381 mvcc.cc:205] Tried to move back new op lower bound from 7022798312584704000 to 7022798311468699648. Current Snapshot: MvccSnapshot[applied={T|T < 7022798312584704000}]
I20240501 07:59:29.307658 1383 mvcc.cc:205] Tried to move back new op lower bound from 7022798312607522816 to 7022798311073583104. Current Snapshot: MvccSnapshot[applied={T|T < 7022798312592134144}]
I20240501 07:59:29.312366 1382 mvcc.cc:205] Tried to move back new op lower bound from 7022798312584704000 to 7022798311468699648. Current Snapshot: MvccSnapshot[applied={T|T < 7022798312584704000}]
I20240501 07:59:29.319943 1388 mvcc.cc:205] Tried to move back new op lower bound from 7022798312584704000 to 7022798311468699648. Current Snapshot: MvccSnapshot[applied={T|T < 7022798312584704000}]
I20240501 07:59:29.331759 1389 mvcc.cc:205] Tried to move back new op lower bound from 7022798312607522816 to 7022798311073583104. Current Snapshot: MvccSnapshot[applied={T|T < 7022798312592134144}]
I20240501 07:59:29.348145 1387 mvcc.cc:205] Tried to move back new op lower bound from 7022798312594079744 to 7022798311070638080. Current Snapshot: MvccSnapshot[applied={T|T < 7022798312594079744}]
INSERT report
rows total: 2000
time total: 138.355 ms
time per row: 0.0691775 ms
I20240501 07:59:29.365677 438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36176:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:29.371310 695 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5764e3fc734f7e9380cb541d5b30e6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e15aa0e6ee4449b2331c23e27e999b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:29.371358 959 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5764e3fc734f7e9380cb541d5b30e6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e15aa0e6ee4449b2331c23e27e999b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:29.371660 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5764e3fc734f7e9380cb541d5b30e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:29.371668 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5764e3fc734f7e9380cb541d5b30e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:29.372090 826 tablet_service.cc:1467] Processing CreateTablet for tablet d123616a55194df79645acca6e1e4ae8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e15aa0e6ee4449b2331c23e27e999b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:29.372359 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d123616a55194df79645acca6e1e4ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:29.373430 1416 tablet_bootstrap.cc:492] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:29.374647 1417 tablet_bootstrap.cc:492] T 1e5764e3fc734f7e9380cb541d5b30e6 P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:29.374743 1090 tablet_service.cc:1467] Processing CreateTablet for tablet d123616a55194df79645acca6e1e4ae8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e15aa0e6ee4449b2331c23e27e999b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:29.374830 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5764e3fc734f7e9380cb541d5b30e6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e15aa0e6ee4449b2331c23e27e999b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:29.375053 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d123616a55194df79645acca6e1e4ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:29.375077 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5764e3fc734f7e9380cb541d5b30e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:29.375254 563 tablet_service.cc:1467] Processing CreateTablet for tablet d123616a55194df79645acca6e1e4ae8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e15aa0e6ee4449b2331c23e27e999b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:29.375571 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d123616a55194df79645acca6e1e4ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:29.375939 1417 tablet_bootstrap.cc:654] T 1e5764e3fc734f7e9380cb541d5b30e6 P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:29.377422 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 07a6844fd48f4ced93631486c4a29489 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e15aa0e6ee4449b2331c23e27e999b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:29.377628 1418 tablet_bootstrap.cc:492] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:29.377688 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a6844fd48f4ced93631486c4a29489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:29.378193 1416 tablet_bootstrap.cc:654] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:29.378654 1418 tablet_bootstrap.cc:654] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:29.378921 1421 tablet_bootstrap.cc:492] T d123616a55194df79645acca6e1e4ae8 P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:29.379778 1421 tablet_bootstrap.cc:654] T d123616a55194df79645acca6e1e4ae8 P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:29.380045 959 tablet_service.cc:1467] Processing CreateTablet for tablet 07a6844fd48f4ced93631486c4a29489 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e15aa0e6ee4449b2331c23e27e999b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:29.380277 1418 tablet_bootstrap.cc:492] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:29.380300 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a6844fd48f4ced93631486c4a29489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:29.380440 1418 ts_tablet_manager.cc:1367] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:29.380786 1421 tablet_bootstrap.cc:492] T d123616a55194df79645acca6e1e4ae8 P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:29.380872 1421 ts_tablet_manager.cc:1367] T d123616a55194df79645acca6e1e4ae8 P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:29.380941 1418 raft_consensus.cc:348] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:29.381052 1418 raft_consensus.cc:374] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:29.381134 1418 raft_consensus.cc:729] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:29.381315 1421 raft_consensus.cc:348] T d123616a55194df79645acca6e1e4ae8 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:29.381407 1421 raft_consensus.cc:374] T d123616a55194df79645acca6e1e4ae8 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:29.381326 1418 consensus_queue.cc:260] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:29.381441 1421 raft_consensus.cc:729] T d123616a55194df79645acca6e1e4ae8 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:29.381564 1421 consensus_queue.cc:260] T d123616a55194df79645acca6e1e4ae8 P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:29.381807 1421 ts_tablet_manager.cc:1398] T d123616a55194df79645acca6e1e4ae8 P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:29.382457 1418 ts_tablet_manager.cc:1398] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:29.382570 1420 tablet_bootstrap.cc:492] T 1e5764e3fc734f7e9380cb541d5b30e6 P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:29.382647 695 tablet_service.cc:1467] Processing CreateTablet for tablet 07a6844fd48f4ced93631486c4a29489 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e15aa0e6ee4449b2331c23e27e999b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:29.382812 1422 tablet_bootstrap.cc:492] T d123616a55194df79645acca6e1e4ae8 P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:29.382895 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a6844fd48f4ced93631486c4a29489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:29.383499 1420 tablet_bootstrap.cc:654] T 1e5764e3fc734f7e9380cb541d5b30e6 P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:29.383792 1422 tablet_bootstrap.cc:654] T d123616a55194df79645acca6e1e4ae8 P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:29.384632 1416 tablet_bootstrap.cc:492] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:29.384717 1416 ts_tablet_manager.cc:1367] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 07:59:29.385388 1416 raft_consensus.cc:348] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.385501 1416 raft_consensus.cc:374] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:29.385537 1416 raft_consensus.cc:729] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:29.385694 1416 consensus_queue.cc:260] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.385973 1416 ts_tablet_manager.cc:1398] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:29.386044 1417 tablet_bootstrap.cc:492] T 1e5764e3fc734f7e9380cb541d5b30e6 P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:29.386140 1417 ts_tablet_manager.cc:1367] T 1e5764e3fc734f7e9380cb541d5b30e6 P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 07:59:29.386184 1416 tablet_bootstrap.cc:492] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:29.386611 1420 tablet_bootstrap.cc:492] T 1e5764e3fc734f7e9380cb541d5b30e6 P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:29.386715 1420 ts_tablet_manager.cc:1367] T 1e5764e3fc734f7e9380cb541d5b30e6 P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:29.386718 1417 raft_consensus.cc:348] T 1e5764e3fc734f7e9380cb541d5b30e6 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.386826 1417 raft_consensus.cc:374] T 1e5764e3fc734f7e9380cb541d5b30e6 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:29.386857 1417 raft_consensus.cc:729] T 1e5764e3fc734f7e9380cb541d5b30e6 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:29.386977 1417 consensus_queue.cc:260] T 1e5764e3fc734f7e9380cb541d5b30e6 P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.387120 1416 tablet_bootstrap.cc:654] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:29.387291 1417 ts_tablet_manager.cc:1398] T 1e5764e3fc734f7e9380cb541d5b30e6 P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:29.387471 1417 tablet_bootstrap.cc:492] T 07a6844fd48f4ced93631486c4a29489 P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:29.387312 1420 raft_consensus.cc:348] T 1e5764e3fc734f7e9380cb541d5b30e6 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.387823 1420 raft_consensus.cc:374] T 1e5764e3fc734f7e9380cb541d5b30e6 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:29.387881 1420 raft_consensus.cc:729] T 1e5764e3fc734f7e9380cb541d5b30e6 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:29.388446 1417 tablet_bootstrap.cc:654] T 07a6844fd48f4ced93631486c4a29489 P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:29.388063 1420 consensus_queue.cc:260] T 1e5764e3fc734f7e9380cb541d5b30e6 P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.389797 1416 tablet_bootstrap.cc:492] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:29.389878 1416 ts_tablet_manager.cc:1367] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:29.390373 1422 tablet_bootstrap.cc:492] T d123616a55194df79645acca6e1e4ae8 P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:29.390353 1416 raft_consensus.cc:348] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.390476 1416 raft_consensus.cc:374] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:29.390527 1416 raft_consensus.cc:729] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:29.390551 1422 ts_tablet_manager.cc:1367] T d123616a55194df79645acca6e1e4ae8 P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:29.390671 1416 consensus_queue.cc:260] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.390977 1416 ts_tablet_manager.cc:1398] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:29.391024 1417 tablet_bootstrap.cc:492] T 07a6844fd48f4ced93631486c4a29489 P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:29.391101 1417 ts_tablet_manager.cc:1367] T 07a6844fd48f4ced93631486c4a29489 P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:29.391201 1422 raft_consensus.cc:348] T d123616a55194df79645acca6e1e4ae8 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:29.391391 1422 raft_consensus.cc:374] T d123616a55194df79645acca6e1e4ae8 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:29.391445 1422 raft_consensus.cc:729] T d123616a55194df79645acca6e1e4ae8 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:29.391553 1417 raft_consensus.cc:348] T 07a6844fd48f4ced93631486c4a29489 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.391657 1417 raft_consensus.cc:374] T 07a6844fd48f4ced93631486c4a29489 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:29.391619 1422 consensus_queue.cc:260] T d123616a55194df79645acca6e1e4ae8 P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:29.391690 1417 raft_consensus.cc:729] T 07a6844fd48f4ced93631486c4a29489 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:29.391810 1417 consensus_queue.cc:260] T 07a6844fd48f4ced93631486c4a29489 P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.391919 1422 ts_tablet_manager.cc:1398] T d123616a55194df79645acca6e1e4ae8 P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:29.392118 1417 ts_tablet_manager.cc:1398] T 07a6844fd48f4ced93631486c4a29489 P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:29.392376 1420 ts_tablet_manager.cc:1398] T 1e5764e3fc734f7e9380cb541d5b30e6 P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:59:29.392566 1420 tablet_bootstrap.cc:492] T 07a6844fd48f4ced93631486c4a29489 P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:29.393909 1420 tablet_bootstrap.cc:654] T 07a6844fd48f4ced93631486c4a29489 P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:29.394369 1386 raft_consensus.cc:482] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:29.394471 1386 raft_consensus.cc:504] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.395167 1420 tablet_bootstrap.cc:492] T 07a6844fd48f4ced93631486c4a29489 P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:29.395267 1420 ts_tablet_manager.cc:1367] T 07a6844fd48f4ced93631486c4a29489 P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:29.395704 1420 raft_consensus.cc:348] T 07a6844fd48f4ced93631486c4a29489 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.395761 1386 leader_election.cc:288] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:29.395807 1420 raft_consensus.cc:374] T 07a6844fd48f4ced93631486c4a29489 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:29.395838 1420 raft_consensus.cc:729] T 07a6844fd48f4ced93631486c4a29489 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:29.395967 1420 consensus_queue.cc:260] T 07a6844fd48f4ced93631486c4a29489 P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.396281 1420 ts_tablet_manager.cc:1398] T 07a6844fd48f4ced93631486c4a29489 P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:29.396622 1391 raft_consensus.cc:482] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:29.396732 1391 raft_consensus.cc:504] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:29.397958 1391 leader_election.cc:288] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:29.401185 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5764e3fc734f7e9380cb541d5b30e6" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429" is_pre_election: true
I20240501 07:59:29.401376 1243 raft_consensus.cc:2454] T 1e5764e3fc734f7e9380cb541d5b30e6 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:29.401744 912 leader_election.cc:302] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:29.402082 1386 raft_consensus.cc:2781] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:29.402154 1386 raft_consensus.cc:482] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:29.402208 1386 raft_consensus.cc:3037] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:29.402688 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5764e3fc734f7e9380cb541d5b30e6" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725" is_pre_election: true
I20240501 07:59:29.402920 715 raft_consensus.cc:2454] T 1e5764e3fc734f7e9380cb541d5b30e6 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:29.404183 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d123616a55194df79645acca6e1e4ae8" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747" is_pre_election: true
I20240501 07:59:29.404332 583 raft_consensus.cc:2454] T d123616a55194df79645acca6e1e4ae8 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:29.404700 781 leader_election.cc:302] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, b9644aeae9de4deba0c3a2b85df04747; no voters:
I20240501 07:59:29.405011 1391 raft_consensus.cc:2781] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:29.405078 1391 raft_consensus.cc:482] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:29.405115 1391 raft_consensus.cc:3037] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:29.405375 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d123616a55194df79645acca6e1e4ae8" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec" is_pre_election: true
I20240501 07:59:29.405509 1106 raft_consensus.cc:2454] T d123616a55194df79645acca6e1e4ae8 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:29.406360 1391 raft_consensus.cc:504] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:29.406721 1391 leader_election.cc:288] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Requested vote from peers b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:29.407064 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d123616a55194df79645acca6e1e4ae8" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747"
I20240501 07:59:29.407214 583 raft_consensus.cc:3037] T d123616a55194df79645acca6e1e4ae8 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:29.407224 1386 raft_consensus.cc:504] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.407634 1386 leader_election.cc:288] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Requested vote from peers 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:29.408133 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5764e3fc734f7e9380cb541d5b30e6" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429"
I20240501 07:59:29.408283 1243 raft_consensus.cc:3037] T 1e5764e3fc734f7e9380cb541d5b30e6 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:29.408356 583 raft_consensus.cc:2454] T d123616a55194df79645acca6e1e4ae8 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:29.408640 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d123616a55194df79645acca6e1e4ae8" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec"
I20240501 07:59:29.408776 1106 raft_consensus.cc:3037] T d123616a55194df79645acca6e1e4ae8 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:29.409847 781 leader_election.cc:302] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, b9644aeae9de4deba0c3a2b85df04747; no voters:
I20240501 07:59:29.410096 1391 raft_consensus.cc:2781] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:29.410187 1391 raft_consensus.cc:686] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee66e185465431389043d933cb51308, State: Running, Role: LEADER
I20240501 07:59:29.410490 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5764e3fc734f7e9380cb541d5b30e6" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725"
I20240501 07:59:29.410511 1391 consensus_queue.cc:237] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:29.410607 715 raft_consensus.cc:3037] T 1e5764e3fc734f7e9380cb541d5b30e6 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:29.413121 437 catalog_manager.cc:5484] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee66e185465431389043d933cb51308 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8ee66e185465431389043d933cb51308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:29.501821 1106 raft_consensus.cc:2454] T d123616a55194df79645acca6e1e4ae8 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:29.501821 715 raft_consensus.cc:2454] T 1e5764e3fc734f7e9380cb541d5b30e6 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:29.501821 1243 raft_consensus.cc:2454] T 1e5764e3fc734f7e9380cb541d5b30e6 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:29.502410 912 leader_election.cc:302] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:29.502671 1386 raft_consensus.cc:2781] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:29.502951 1386 raft_consensus.cc:686] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [term 1 LEADER]: Becoming Leader. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Running, Role: LEADER
I20240501 07:59:29.503329 1386 consensus_queue.cc:237] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.506335 437 catalog_manager.cc:5484] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1526c5008f714bcfa69e9bacb51a662c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:29.594906 1386 raft_consensus.cc:482] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:29.595052 1386 raft_consensus.cc:504] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.595492 1386 leader_election.cc:288] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:29.596067 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a6844fd48f4ced93631486c4a29489" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725" is_pre_election: true
I20240501 07:59:29.596033 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a6844fd48f4ced93631486c4a29489" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429" is_pre_election: true
I20240501 07:59:29.596225 715 raft_consensus.cc:2454] T 07a6844fd48f4ced93631486c4a29489 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:29.596226 1243 raft_consensus.cc:2454] T 07a6844fd48f4ced93631486c4a29489 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:29.596560 913 leader_election.cc:302] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, 3ee011df676c43f7beaeace0f8c60725; no voters:
I20240501 07:59:29.596807 1386 raft_consensus.cc:2781] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:29.596887 1386 raft_consensus.cc:482] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:29.596935 1386 raft_consensus.cc:3037] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:29.598265 1386 raft_consensus.cc:504] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.598754 1386 leader_election.cc:288] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Requested vote from peers 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:29.599184 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a6844fd48f4ced93631486c4a29489" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725"
I20240501 07:59:29.599211 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a6844fd48f4ced93631486c4a29489" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429"
I20240501 07:59:29.599346 1243 raft_consensus.cc:3037] T 07a6844fd48f4ced93631486c4a29489 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:29.599330 715 raft_consensus.cc:3037] T 07a6844fd48f4ced93631486c4a29489 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:29.695778 1243 raft_consensus.cc:2454] T 07a6844fd48f4ced93631486c4a29489 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:29.695819 715 raft_consensus.cc:2454] T 07a6844fd48f4ced93631486c4a29489 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:29.696251 912 leader_election.cc:302] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:29.696467 1386 raft_consensus.cc:2781] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:29.696564 1386 raft_consensus.cc:686] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [term 1 LEADER]: Becoming Leader. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Running, Role: LEADER
I20240501 07:59:29.696743 1386 consensus_queue.cc:237] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:29.698896 437 catalog_manager.cc:5484] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1526c5008f714bcfa69e9bacb51a662c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:29.823822 1384 consensus_queue.cc:1035] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:29.827929 1384 consensus_queue.cc:1035] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:59:29.867779 1440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:29.868077 1440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:29.871460 1440 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 07:59:29.891125 1386 consensus_queue.cc:1035] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240501 07:59:29.896847 1386 consensus_queue.cc:1035] T 1e5764e3fc734f7e9380cb541d5b30e6 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:29.977537 715 raft_consensus.cc:1261] T 07a6844fd48f4ced93631486c4a29489 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Refusing update from remote peer 1526c5008f714bcfa69e9bacb51a662c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:29.977536 1243 raft_consensus.cc:1261] T 07a6844fd48f4ced93631486c4a29489 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Refusing update from remote peer 1526c5008f714bcfa69e9bacb51a662c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:29.978042 1437 consensus_queue.cc:1035] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:29.983801 1386 consensus_queue.cc:1035] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 116.636 ms
time per row: 0.058318 ms
I20240501 07:59:30.042374 434 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36176:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:30.048148 563 tablet_service.cc:1467] Processing CreateTablet for tablet a523579b0ba14640bfbd24d4fd2f5606 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f57e24e1ef8c40a89fa9085446d3be34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.048203 1090 tablet_service.cc:1467] Processing CreateTablet for tablet a523579b0ba14640bfbd24d4fd2f5606 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f57e24e1ef8c40a89fa9085446d3be34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.048496 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a523579b0ba14640bfbd24d4fd2f5606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.048514 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a523579b0ba14640bfbd24d4fd2f5606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.048519 959 tablet_service.cc:1467] Processing CreateTablet for tablet 87d19c5333d448c0aa8440fff8ded2ef (DEFAULT_TABLE table=rebalance_test_table_2 [id=f57e24e1ef8c40a89fa9085446d3be34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.048750 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87d19c5333d448c0aa8440fff8ded2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.049340 695 tablet_service.cc:1467] Processing CreateTablet for tablet 87d19c5333d448c0aa8440fff8ded2ef (DEFAULT_TABLE table=rebalance_test_table_2 [id=f57e24e1ef8c40a89fa9085446d3be34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.049623 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87d19c5333d448c0aa8440fff8ded2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.051066 826 tablet_service.cc:1467] Processing CreateTablet for tablet a523579b0ba14640bfbd24d4fd2f5606 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f57e24e1ef8c40a89fa9085446d3be34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.051337 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a523579b0ba14640bfbd24d4fd2f5606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.051616 827 tablet_service.cc:1467] Processing CreateTablet for tablet 85220f32c33f4b7085852164e6b19add (DEFAULT_TABLE table=rebalance_test_table_2 [id=f57e24e1ef8c40a89fa9085446d3be34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.051852 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85220f32c33f4b7085852164e6b19add. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.051965 1501 tablet_bootstrap.cc:492] T a523579b0ba14640bfbd24d4fd2f5606 P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:30.052901 1501 tablet_bootstrap.cc:654] T a523579b0ba14640bfbd24d4fd2f5606 P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.053527 1502 tablet_bootstrap.cc:492] T 87d19c5333d448c0aa8440fff8ded2ef P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:30.053635 1503 tablet_bootstrap.cc:492] T 87d19c5333d448c0aa8440fff8ded2ef P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:30.054517 1502 tablet_bootstrap.cc:654] T 87d19c5333d448c0aa8440fff8ded2ef P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.054517 1503 tablet_bootstrap.cc:654] T 87d19c5333d448c0aa8440fff8ded2ef P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.054602 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 87d19c5333d448c0aa8440fff8ded2ef (DEFAULT_TABLE table=rebalance_test_table_2 [id=f57e24e1ef8c40a89fa9085446d3be34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.054608 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 85220f32c33f4b7085852164e6b19add (DEFAULT_TABLE table=rebalance_test_table_2 [id=f57e24e1ef8c40a89fa9085446d3be34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.054883 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85220f32c33f4b7085852164e6b19add. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.054883 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87d19c5333d448c0aa8440fff8ded2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.055361 1505 tablet_bootstrap.cc:492] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:30.055737 1501 tablet_bootstrap.cc:492] T a523579b0ba14640bfbd24d4fd2f5606 P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:30.055856 1501 ts_tablet_manager.cc:1367] T a523579b0ba14640bfbd24d4fd2f5606 P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:30.056378 1505 tablet_bootstrap.cc:654] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.056349 1501 raft_consensus.cc:348] T a523579b0ba14640bfbd24d4fd2f5606 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.056483 1501 raft_consensus.cc:374] T a523579b0ba14640bfbd24d4fd2f5606 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.056529 1501 raft_consensus.cc:729] T a523579b0ba14640bfbd24d4fd2f5606 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.056723 1501 consensus_queue.cc:260] T a523579b0ba14640bfbd24d4fd2f5606 P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.057446 563 tablet_service.cc:1467] Processing CreateTablet for tablet 85220f32c33f4b7085852164e6b19add (DEFAULT_TABLE table=rebalance_test_table_2 [id=f57e24e1ef8c40a89fa9085446d3be34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.057549 1501 ts_tablet_manager.cc:1398] T a523579b0ba14640bfbd24d4fd2f5606 P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:30.057687 1502 tablet_bootstrap.cc:492] T 87d19c5333d448c0aa8440fff8ded2ef P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:30.057718 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85220f32c33f4b7085852164e6b19add. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.057801 1502 ts_tablet_manager.cc:1367] T 87d19c5333d448c0aa8440fff8ded2ef P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:30.058254 1502 raft_consensus.cc:348] T 87d19c5333d448c0aa8440fff8ded2ef P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:30.058440 1502 raft_consensus.cc:374] T 87d19c5333d448c0aa8440fff8ded2ef P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.058494 1502 raft_consensus.cc:729] T 87d19c5333d448c0aa8440fff8ded2ef P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.058676 1502 consensus_queue.cc:260] T 87d19c5333d448c0aa8440fff8ded2ef P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:30.058929 1502 ts_tablet_manager.cc:1398] T 87d19c5333d448c0aa8440fff8ded2ef P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:30.060144 1503 tablet_bootstrap.cc:492] T 87d19c5333d448c0aa8440fff8ded2ef P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:30.060144 1501 tablet_bootstrap.cc:492] T 85220f32c33f4b7085852164e6b19add P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:30.060252 1503 ts_tablet_manager.cc:1367] T 87d19c5333d448c0aa8440fff8ded2ef P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 07:59:30.060249 1500 tablet_bootstrap.cc:492] T a523579b0ba14640bfbd24d4fd2f5606 P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:30.060743 1503 raft_consensus.cc:348] T 87d19c5333d448c0aa8440fff8ded2ef P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:30.060887 1503 raft_consensus.cc:374] T 87d19c5333d448c0aa8440fff8ded2ef P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.060926 1503 raft_consensus.cc:729] T 87d19c5333d448c0aa8440fff8ded2ef P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.061112 1503 consensus_queue.cc:260] T 87d19c5333d448c0aa8440fff8ded2ef P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:30.061247 1500 tablet_bootstrap.cc:654] T a523579b0ba14640bfbd24d4fd2f5606 P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.061436 1508 tablet_bootstrap.cc:492] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:30.061720 1501 tablet_bootstrap.cc:654] T 85220f32c33f4b7085852164e6b19add P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.062978 1501 tablet_bootstrap.cc:492] T 85220f32c33f4b7085852164e6b19add P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:30.063063 1505 tablet_bootstrap.cc:492] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:30.063066 1501 ts_tablet_manager.cc:1367] T 85220f32c33f4b7085852164e6b19add P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:30.063145 1505 ts_tablet_manager.cc:1367] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:30.063417 1508 tablet_bootstrap.cc:654] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.063519 1501 raft_consensus.cc:348] T 85220f32c33f4b7085852164e6b19add P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.063623 1501 raft_consensus.cc:374] T 85220f32c33f4b7085852164e6b19add P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.063655 1501 raft_consensus.cc:729] T 85220f32c33f4b7085852164e6b19add P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.063716 1505 raft_consensus.cc:348] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.063812 1505 raft_consensus.cc:374] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.063845 1505 raft_consensus.cc:729] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.063813 1501 consensus_queue.cc:260] T 85220f32c33f4b7085852164e6b19add P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.063959 1505 consensus_queue.cc:260] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.064095 1501 ts_tablet_manager.cc:1398] T 85220f32c33f4b7085852164e6b19add P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:30.064307 1505 ts_tablet_manager.cc:1398] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:30.064468 1505 tablet_bootstrap.cc:492] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:30.064757 1500 tablet_bootstrap.cc:492] T a523579b0ba14640bfbd24d4fd2f5606 P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:30.064875 1500 ts_tablet_manager.cc:1367] T a523579b0ba14640bfbd24d4fd2f5606 P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:30.065404 1505 tablet_bootstrap.cc:654] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.066355 1500 raft_consensus.cc:348] T a523579b0ba14640bfbd24d4fd2f5606 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.066493 1500 raft_consensus.cc:374] T a523579b0ba14640bfbd24d4fd2f5606 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.066536 1500 raft_consensus.cc:729] T a523579b0ba14640bfbd24d4fd2f5606 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.066612 1505 tablet_bootstrap.cc:492] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:30.066690 1505 ts_tablet_manager.cc:1367] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:59:30.066685 1500 consensus_queue.cc:260] T a523579b0ba14640bfbd24d4fd2f5606 P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.067006 1500 ts_tablet_manager.cc:1398] T a523579b0ba14640bfbd24d4fd2f5606 P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:30.067117 1505 raft_consensus.cc:348] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.067232 1505 raft_consensus.cc:374] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.067267 1505 raft_consensus.cc:729] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.067615 1500 tablet_bootstrap.cc:492] T 85220f32c33f4b7085852164e6b19add P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:30.068679 1500 tablet_bootstrap.cc:654] T 85220f32c33f4b7085852164e6b19add P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.068914 1505 consensus_queue.cc:260] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.069394 1505 ts_tablet_manager.cc:1398] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:59:30.070194 1500 tablet_bootstrap.cc:492] T 85220f32c33f4b7085852164e6b19add P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:30.070304 1500 ts_tablet_manager.cc:1367] T 85220f32c33f4b7085852164e6b19add P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:30.070710 1500 raft_consensus.cc:348] T 85220f32c33f4b7085852164e6b19add P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.070802 1500 raft_consensus.cc:374] T 85220f32c33f4b7085852164e6b19add P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.070835 1500 raft_consensus.cc:729] T 85220f32c33f4b7085852164e6b19add P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.070968 1500 consensus_queue.cc:260] T 85220f32c33f4b7085852164e6b19add P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.071261 1500 ts_tablet_manager.cc:1398] T 85220f32c33f4b7085852164e6b19add P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:30.071671 1503 ts_tablet_manager.cc:1398] T 87d19c5333d448c0aa8440fff8ded2ef P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240501 07:59:30.074337 1508 tablet_bootstrap.cc:492] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:30.074456 1508 ts_tablet_manager.cc:1367] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:59:30.074973 1508 raft_consensus.cc:348] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:30.075089 1508 raft_consensus.cc:374] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.075124 1508 raft_consensus.cc:729] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.075253 1508 consensus_queue.cc:260] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:30.075542 1508 ts_tablet_manager.cc:1398] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:30.176514 1480 raft_consensus.cc:482] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:30.176673 1480 raft_consensus.cc:504] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.177178 1480 leader_election.cc:288] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665), b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609)
I20240501 07:59:30.177690 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85220f32c33f4b7085852164e6b19add" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec" is_pre_election: true
I20240501 07:59:30.177692 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85220f32c33f4b7085852164e6b19add" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747" is_pre_election: true
I20240501 07:59:30.177825 1106 raft_consensus.cc:2454] T 85220f32c33f4b7085852164e6b19add P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:30.177901 583 raft_consensus.cc:2454] T 85220f32c33f4b7085852164e6b19add P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:30.178246 780 leader_election.cc:302] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:30.178442 1480 raft_consensus.cc:2781] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:30.178501 1480 raft_consensus.cc:482] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:30.178553 1480 raft_consensus.cc:3037] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:30.181034 1480 raft_consensus.cc:504] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.181437 1480 leader_election.cc:288] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Requested vote from peers f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665), b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609)
I20240501 07:59:30.181912 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85220f32c33f4b7085852164e6b19add" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec"
I20240501 07:59:30.181912 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85220f32c33f4b7085852164e6b19add" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747"
I20240501 07:59:30.182070 583 raft_consensus.cc:3037] T 85220f32c33f4b7085852164e6b19add P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:30.182070 1106 raft_consensus.cc:3037] T 85220f32c33f4b7085852164e6b19add P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:30.183059 583 raft_consensus.cc:2454] T 85220f32c33f4b7085852164e6b19add P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:30.183051 1106 raft_consensus.cc:2454] T 85220f32c33f4b7085852164e6b19add P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:30.183360 780 leader_election.cc:302] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:30.183553 1480 raft_consensus.cc:2781] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:30.183683 1480 raft_consensus.cc:686] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee66e185465431389043d933cb51308, State: Running, Role: LEADER
I20240501 07:59:30.183857 1480 consensus_queue.cc:237] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.186662 432 catalog_manager.cc:5484] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee66e185465431389043d933cb51308 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8ee66e185465431389043d933cb51308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:30.351405 1465 raft_consensus.cc:482] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:30.351568 1465 raft_consensus.cc:504] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:30.352082 1480 raft_consensus.cc:482] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:30.352207 1480 raft_consensus.cc:504] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.352708 1480 leader_election.cc:288] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:30.352882 1465 leader_election.cc:288] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765), 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663)
I20240501 07:59:30.353317 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a523579b0ba14640bfbd24d4fd2f5606" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747" is_pre_election: true
I20240501 07:59:30.353358 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a523579b0ba14640bfbd24d4fd2f5606" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec" is_pre_election: true
I20240501 07:59:30.353463 583 raft_consensus.cc:2454] T a523579b0ba14640bfbd24d4fd2f5606 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:30.353472 1106 raft_consensus.cc:2454] T a523579b0ba14640bfbd24d4fd2f5606 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:30.353776 780 leader_election.cc:302] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:30.354199 1480 raft_consensus.cc:2781] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:30.354259 1480 raft_consensus.cc:482] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:30.354301 1480 raft_consensus.cc:3037] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:30.355357 1480 raft_consensus.cc:504] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.355726 1480 leader_election.cc:288] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Requested vote from peers b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:30.356444 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a523579b0ba14640bfbd24d4fd2f5606" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747"
I20240501 07:59:30.356568 583 raft_consensus.cc:3037] T a523579b0ba14640bfbd24d4fd2f5606 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:30.356621 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a523579b0ba14640bfbd24d4fd2f5606" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec"
I20240501 07:59:30.356741 1106 raft_consensus.cc:3037] T a523579b0ba14640bfbd24d4fd2f5606 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:30.358592 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d19c5333d448c0aa8440fff8ded2ef" candidate_uuid: "c45c306768564fb2bd741c535d334429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1526c5008f714bcfa69e9bacb51a662c" is_pre_election: true
I20240501 07:59:30.358729 979 raft_consensus.cc:2454] T 87d19c5333d448c0aa8440fff8ded2ef P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c45c306768564fb2bd741c535d334429 in term 0.
I20240501 07:59:30.359086 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d19c5333d448c0aa8440fff8ded2ef" candidate_uuid: "c45c306768564fb2bd741c535d334429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725" is_pre_election: true
I20240501 07:59:30.359125 1177 leader_election.cc:302] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:30.359225 715 raft_consensus.cc:2454] T 87d19c5333d448c0aa8440fff8ded2ef P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c45c306768564fb2bd741c535d334429 in term 0.
I20240501 07:59:30.359359 1465 raft_consensus.cc:2781] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:30.359416 1465 raft_consensus.cc:482] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:30.359462 1465 raft_consensus.cc:3037] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:30.438357 583 raft_consensus.cc:2454] T a523579b0ba14640bfbd24d4fd2f5606 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:30.438357 1106 raft_consensus.cc:2454] T a523579b0ba14640bfbd24d4fd2f5606 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:30.438364 1465 raft_consensus.cc:504] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:30.438792 781 leader_election.cc:302] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, b9644aeae9de4deba0c3a2b85df04747; no voters:
I20240501 07:59:30.438886 1465 leader_election.cc:288] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [CANDIDATE]: Term 1 election: Requested vote from peers 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765), 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663)
I20240501 07:59:30.439165 1480 raft_consensus.cc:2781] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:30.439268 1480 raft_consensus.cc:686] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee66e185465431389043d933cb51308, State: Running, Role: LEADER
I20240501 07:59:30.439332 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d19c5333d448c0aa8440fff8ded2ef" candidate_uuid: "c45c306768564fb2bd741c535d334429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725"
I20240501 07:59:30.439360 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d19c5333d448c0aa8440fff8ded2ef" candidate_uuid: "c45c306768564fb2bd741c535d334429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1526c5008f714bcfa69e9bacb51a662c"
I20240501 07:59:30.439409 1480 consensus_queue.cc:237] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:30.439471 715 raft_consensus.cc:3037] T 87d19c5333d448c0aa8440fff8ded2ef P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:30.439487 979 raft_consensus.cc:3037] T 87d19c5333d448c0aa8440fff8ded2ef P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:30.440564 979 raft_consensus.cc:2454] T 87d19c5333d448c0aa8440fff8ded2ef P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c45c306768564fb2bd741c535d334429 in term 1.
I20240501 07:59:30.440722 715 raft_consensus.cc:2454] T 87d19c5333d448c0aa8440fff8ded2ef P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c45c306768564fb2bd741c535d334429 in term 1.
I20240501 07:59:30.440944 1177 leader_election.cc:302] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:30.441123 1465 raft_consensus.cc:2781] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:30.441291 1465 raft_consensus.cc:686] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [term 1 LEADER]: Becoming Leader. State: Replica: c45c306768564fb2bd741c535d334429, State: Running, Role: LEADER
I20240501 07:59:30.441653 1465 consensus_queue.cc:237] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:30.441679 432 catalog_manager.cc:5484] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee66e185465431389043d933cb51308 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8ee66e185465431389043d933cb51308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:30.444205 434 catalog_manager.cc:5484] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 reported cstate change: term changed from 0 to 1, leader changed from <none> to c45c306768564fb2bd741c535d334429 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "c45c306768564fb2bd741c535d334429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: HEALTHY } } }
W20240501 07:59:30.635968 1522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:30.636258 1522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 07:59:30.638510 1480 consensus_queue.cc:1035] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:59:30.640153 1522 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 07:59:30.642856 1480 consensus_queue.cc:1035] T 85220f32c33f4b7085852164e6b19add P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 07:59:30.722532 1106 raft_consensus.cc:1261] T a523579b0ba14640bfbd24d4fd2f5606 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Refusing update from remote peer 8ee66e185465431389043d933cb51308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:30.723059 1480 consensus_queue.cc:1035] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:30.723729 583 raft_consensus.cc:1261] T a523579b0ba14640bfbd24d4fd2f5606 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Refusing update from remote peer 8ee66e185465431389043d933cb51308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:30.724532 1560 consensus_queue.cc:1035] T a523579b0ba14640bfbd24d4fd2f5606 P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:30.734643 715 raft_consensus.cc:1261] T 87d19c5333d448c0aa8440fff8ded2ef P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Refusing update from remote peer c45c306768564fb2bd741c535d334429: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:30.734884 979 raft_consensus.cc:1261] T 87d19c5333d448c0aa8440fff8ded2ef P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Refusing update from remote peer c45c306768564fb2bd741c535d334429: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:30.737059 1465 consensus_queue.cc:1035] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:30.738248 1520 consensus_queue.cc:1035] T 87d19c5333d448c0aa8440fff8ded2ef P c45c306768564fb2bd741c535d334429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 109.17 ms
time per row: 0.0545848 ms
I20240501 07:59:30.796921 432 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36176:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:30.803021 959 tablet_service.cc:1467] Processing CreateTablet for tablet 549c8b9eef0440979653e95da1523966 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d2c55d15deed47a9beb96ee83b19c30f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.803097 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 549c8b9eef0440979653e95da1523966 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d2c55d15deed47a9beb96ee83b19c30f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.803622 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549c8b9eef0440979653e95da1523966. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.803647 695 tablet_service.cc:1467] Processing CreateTablet for tablet 549c8b9eef0440979653e95da1523966 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d2c55d15deed47a9beb96ee83b19c30f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.803676 826 tablet_service.cc:1467] Processing CreateTablet for tablet dc96c465fe554a94aa599200d717a3c2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d2c55d15deed47a9beb96ee83b19c30f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.803838 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 5240af8639074b7b92c38021d56b3482 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d2c55d15deed47a9beb96ee83b19c30f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.803928 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549c8b9eef0440979653e95da1523966. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.804064 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5240af8639074b7b92c38021d56b3482. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.804085 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc96c465fe554a94aa599200d717a3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.806212 1586 tablet_bootstrap.cc:492] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:30.806818 563 tablet_service.cc:1467] Processing CreateTablet for tablet dc96c465fe554a94aa599200d717a3c2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d2c55d15deed47a9beb96ee83b19c30f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.806985 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549c8b9eef0440979653e95da1523966. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.807109 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc96c465fe554a94aa599200d717a3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.807214 1586 tablet_bootstrap.cc:654] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.807520 1588 tablet_bootstrap.cc:492] T dc96c465fe554a94aa599200d717a3c2 P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:30.807905 1587 tablet_bootstrap.cc:492] T 5240af8639074b7b92c38021d56b3482 P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:30.808432 1588 tablet_bootstrap.cc:654] T dc96c465fe554a94aa599200d717a3c2 P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.808853 1587 tablet_bootstrap.cc:654] T 5240af8639074b7b92c38021d56b3482 P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.809417 1591 tablet_bootstrap.cc:492] T dc96c465fe554a94aa599200d717a3c2 P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:30.809531 1090 tablet_service.cc:1467] Processing CreateTablet for tablet dc96c465fe554a94aa599200d717a3c2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d2c55d15deed47a9beb96ee83b19c30f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.809573 695 tablet_service.cc:1467] Processing CreateTablet for tablet 5240af8639074b7b92c38021d56b3482 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d2c55d15deed47a9beb96ee83b19c30f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.809808 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5240af8639074b7b92c38021d56b3482. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.809819 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc96c465fe554a94aa599200d717a3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.810308 1591 tablet_bootstrap.cc:654] T dc96c465fe554a94aa599200d717a3c2 P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.810834 959 tablet_service.cc:1467] Processing CreateTablet for tablet 5240af8639074b7b92c38021d56b3482 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d2c55d15deed47a9beb96ee83b19c30f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:30.811098 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5240af8639074b7b92c38021d56b3482. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:30.811369 1591 tablet_bootstrap.cc:492] T dc96c465fe554a94aa599200d717a3c2 P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:30.811441 1591 ts_tablet_manager.cc:1367] T dc96c465fe554a94aa599200d717a3c2 P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:30.811897 1591 raft_consensus.cc:348] T dc96c465fe554a94aa599200d717a3c2 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:30.812024 1591 raft_consensus.cc:374] T dc96c465fe554a94aa599200d717a3c2 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.812062 1591 raft_consensus.cc:729] T dc96c465fe554a94aa599200d717a3c2 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.812186 1591 consensus_queue.cc:260] T dc96c465fe554a94aa599200d717a3c2 P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:30.812450 1591 ts_tablet_manager.cc:1398] T dc96c465fe554a94aa599200d717a3c2 P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:30.812867 1585 tablet_bootstrap.cc:492] T 549c8b9eef0440979653e95da1523966 P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:30.813948 1585 tablet_bootstrap.cc:654] T 549c8b9eef0440979653e95da1523966 P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.814388 1588 tablet_bootstrap.cc:492] T dc96c465fe554a94aa599200d717a3c2 P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:30.814559 1588 ts_tablet_manager.cc:1367] T dc96c465fe554a94aa599200d717a3c2 P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:30.815009 1594 tablet_bootstrap.cc:492] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:30.815086 1585 tablet_bootstrap.cc:492] T 549c8b9eef0440979653e95da1523966 P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:30.815169 1585 ts_tablet_manager.cc:1367] T 549c8b9eef0440979653e95da1523966 P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:30.815433 1588 raft_consensus.cc:348] T dc96c465fe554a94aa599200d717a3c2 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:30.815590 1588 raft_consensus.cc:374] T dc96c465fe554a94aa599200d717a3c2 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.815630 1588 raft_consensus.cc:729] T dc96c465fe554a94aa599200d717a3c2 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.815588 1585 raft_consensus.cc:348] T 549c8b9eef0440979653e95da1523966 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:30.815683 1585 raft_consensus.cc:374] T 549c8b9eef0440979653e95da1523966 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.815716 1585 raft_consensus.cc:729] T 549c8b9eef0440979653e95da1523966 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.815822 1588 consensus_queue.cc:260] T dc96c465fe554a94aa599200d717a3c2 P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:30.815843 1585 consensus_queue.cc:260] T 549c8b9eef0440979653e95da1523966 P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:30.815853 1594 tablet_bootstrap.cc:654] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.816087 1585 ts_tablet_manager.cc:1398] T 549c8b9eef0440979653e95da1523966 P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:30.816248 1585 tablet_bootstrap.cc:492] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:30.817121 1585 tablet_bootstrap.cc:654] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.817683 1588 ts_tablet_manager.cc:1398] T dc96c465fe554a94aa599200d717a3c2 P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:30.818477 1585 tablet_bootstrap.cc:492] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:30.818560 1585 ts_tablet_manager.cc:1367] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:30.819008 1585 raft_consensus.cc:348] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:30.819079 1586 tablet_bootstrap.cc:492] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:30.819108 1585 raft_consensus.cc:374] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.819146 1585 raft_consensus.cc:729] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.819192 1586 ts_tablet_manager.cc:1367] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:59:30.819269 1585 consensus_queue.cc:260] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:30.819556 1585 ts_tablet_manager.cc:1398] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:30.819809 1586 raft_consensus.cc:348] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:30.819881 1594 tablet_bootstrap.cc:492] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:30.819934 1586 raft_consensus.cc:374] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.819965 1586 raft_consensus.cc:729] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.819983 1594 ts_tablet_manager.cc:1367] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:30.820165 1586 consensus_queue.cc:260] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:30.820611 1594 raft_consensus.cc:348] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:30.820791 1594 raft_consensus.cc:374] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.820885 1594 raft_consensus.cc:729] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.821105 1594 consensus_queue.cc:260] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:30.821250 1586 ts_tablet_manager.cc:1398] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:30.821420 1586 tablet_bootstrap.cc:492] T 5240af8639074b7b92c38021d56b3482 P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:30.821461 1594 ts_tablet_manager.cc:1398] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:30.822501 1587 tablet_bootstrap.cc:492] T 5240af8639074b7b92c38021d56b3482 P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:30.822669 1587 ts_tablet_manager.cc:1367] T 5240af8639074b7b92c38021d56b3482 P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 07:59:30.822820 1586 tablet_bootstrap.cc:654] T 5240af8639074b7b92c38021d56b3482 P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.823413 1587 raft_consensus.cc:348] T 5240af8639074b7b92c38021d56b3482 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:30.823575 1587 raft_consensus.cc:374] T 5240af8639074b7b92c38021d56b3482 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.823642 1587 raft_consensus.cc:729] T 5240af8639074b7b92c38021d56b3482 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.823854 1587 consensus_queue.cc:260] T 5240af8639074b7b92c38021d56b3482 P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:30.824198 1586 tablet_bootstrap.cc:492] T 5240af8639074b7b92c38021d56b3482 P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:30.824199 1587 ts_tablet_manager.cc:1398] T 5240af8639074b7b92c38021d56b3482 P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:30.824285 1586 ts_tablet_manager.cc:1367] T 5240af8639074b7b92c38021d56b3482 P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:30.824393 1587 tablet_bootstrap.cc:492] T 549c8b9eef0440979653e95da1523966 P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:30.824781 1586 raft_consensus.cc:348] T 5240af8639074b7b92c38021d56b3482 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:30.824878 1586 raft_consensus.cc:374] T 5240af8639074b7b92c38021d56b3482 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.824906 1586 raft_consensus.cc:729] T 5240af8639074b7b92c38021d56b3482 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.825058 1586 consensus_queue.cc:260] T 5240af8639074b7b92c38021d56b3482 P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:30.825316 1586 ts_tablet_manager.cc:1398] T 5240af8639074b7b92c38021d56b3482 P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:30.825701 1587 tablet_bootstrap.cc:654] T 549c8b9eef0440979653e95da1523966 P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:30.827116 1587 tablet_bootstrap.cc:492] T 549c8b9eef0440979653e95da1523966 P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:30.827211 1587 ts_tablet_manager.cc:1367] T 549c8b9eef0440979653e95da1523966 P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:30.827571 1587 raft_consensus.cc:348] T 549c8b9eef0440979653e95da1523966 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:30.827679 1587 raft_consensus.cc:374] T 549c8b9eef0440979653e95da1523966 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:30.827713 1587 raft_consensus.cc:729] T 549c8b9eef0440979653e95da1523966 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:30.827910 1587 consensus_queue.cc:260] T 549c8b9eef0440979653e95da1523966 P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:30.828153 1587 ts_tablet_manager.cc:1398] T 549c8b9eef0440979653e95da1523966 P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:30.834570 1346 raft_consensus.cc:482] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:30.834692 1346 raft_consensus.cc:504] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:30.835234 1346 leader_election.cc:288] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c45c306768564fb2bd741c535d334429 (127.0.99.134:37067), 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663)
I20240501 07:59:30.835709 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549c8b9eef0440979653e95da1523966" candidate_uuid: "3ee011df676c43f7beaeace0f8c60725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1526c5008f714bcfa69e9bacb51a662c" is_pre_election: true
I20240501 07:59:30.835884 979 raft_consensus.cc:2454] T 549c8b9eef0440979653e95da1523966 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ee011df676c43f7beaeace0f8c60725 in term 0.
I20240501 07:59:30.836333 649 leader_election.cc:302] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, 3ee011df676c43f7beaeace0f8c60725; no voters:
I20240501 07:59:30.836539 1346 raft_consensus.cc:2781] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:30.836609 1346 raft_consensus.cc:482] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:30.836664 1346 raft_consensus.cc:3037] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:30.839406 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549c8b9eef0440979653e95da1523966" candidate_uuid: "3ee011df676c43f7beaeace0f8c60725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429" is_pre_election: true
I20240501 07:59:30.839565 1243 raft_consensus.cc:2454] T 549c8b9eef0440979653e95da1523966 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ee011df676c43f7beaeace0f8c60725 in term 0.
I20240501 07:59:30.853950 1346 raft_consensus.cc:504] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:30.854476 1346 leader_election.cc:288] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [CANDIDATE]: Term 1 election: Requested vote from peers c45c306768564fb2bd741c535d334429 (127.0.99.134:37067), 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663)
I20240501 07:59:30.855011 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549c8b9eef0440979653e95da1523966" candidate_uuid: "3ee011df676c43f7beaeace0f8c60725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1526c5008f714bcfa69e9bacb51a662c"
I20240501 07:59:30.855023 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549c8b9eef0440979653e95da1523966" candidate_uuid: "3ee011df676c43f7beaeace0f8c60725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429"
I20240501 07:59:30.855162 979 raft_consensus.cc:3037] T 549c8b9eef0440979653e95da1523966 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:30.855177 1243 raft_consensus.cc:3037] T 549c8b9eef0440979653e95da1523966 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:30.856313 979 raft_consensus.cc:2454] T 549c8b9eef0440979653e95da1523966 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ee011df676c43f7beaeace0f8c60725 in term 1.
I20240501 07:59:30.856422 1243 raft_consensus.cc:2454] T 549c8b9eef0440979653e95da1523966 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ee011df676c43f7beaeace0f8c60725 in term 1.
I20240501 07:59:30.856887 649 leader_election.cc:302] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, 3ee011df676c43f7beaeace0f8c60725; no voters:
I20240501 07:59:30.857177 1346 raft_consensus.cc:2781] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:30.857426 1346 raft_consensus.cc:686] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [term 1 LEADER]: Becoming Leader. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Running, Role: LEADER
I20240501 07:59:30.857671 1346 consensus_queue.cc:237] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:30.860085 439 catalog_manager.cc:5484] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3ee011df676c43f7beaeace0f8c60725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:30.940623 1320 raft_consensus.cc:482] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:30.940771 1320 raft_consensus.cc:504] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:30.941287 1320 leader_election.cc:288] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609), 8ee66e185465431389043d933cb51308 (127.0.99.131:37541)
I20240501 07:59:30.941735 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc96c465fe554a94aa599200d717a3c2" candidate_uuid: "f5af98ef61b54ae0b33faeb78af96dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747" is_pre_election: true
I20240501 07:59:30.941732 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc96c465fe554a94aa599200d717a3c2" candidate_uuid: "f5af98ef61b54ae0b33faeb78af96dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308" is_pre_election: true
I20240501 07:59:30.941874 583 raft_consensus.cc:2454] T dc96c465fe554a94aa599200d717a3c2 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5af98ef61b54ae0b33faeb78af96dec in term 0.
I20240501 07:59:30.941900 846 raft_consensus.cc:2454] T dc96c465fe554a94aa599200d717a3c2 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5af98ef61b54ae0b33faeb78af96dec in term 0.
I20240501 07:59:30.942247 1045 leader_election.cc:302] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9644aeae9de4deba0c3a2b85df04747, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:30.942494 1320 raft_consensus.cc:2781] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:30.942551 1320 raft_consensus.cc:482] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:30.942589 1320 raft_consensus.cc:3037] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:30.952796 1320 raft_consensus.cc:504] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:30.953398 1320 leader_election.cc:288] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [CANDIDATE]: Term 1 election: Requested vote from peers b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609), 8ee66e185465431389043d933cb51308 (127.0.99.131:37541)
I20240501 07:59:30.953939 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc96c465fe554a94aa599200d717a3c2" candidate_uuid: "f5af98ef61b54ae0b33faeb78af96dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308"
I20240501 07:59:30.953938 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc96c465fe554a94aa599200d717a3c2" candidate_uuid: "f5af98ef61b54ae0b33faeb78af96dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747"
I20240501 07:59:30.954115 846 raft_consensus.cc:3037] T dc96c465fe554a94aa599200d717a3c2 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:30.954128 583 raft_consensus.cc:3037] T dc96c465fe554a94aa599200d717a3c2 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:30.954916 583 raft_consensus.cc:2454] T dc96c465fe554a94aa599200d717a3c2 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5af98ef61b54ae0b33faeb78af96dec in term 1.
I20240501 07:59:30.955279 1045 leader_election.cc:302] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9644aeae9de4deba0c3a2b85df04747, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:30.955471 1320 raft_consensus.cc:2781] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:30.955651 1320 raft_consensus.cc:686] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [term 1 LEADER]: Becoming Leader. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Running, Role: LEADER
I20240501 07:59:30.955806 846 raft_consensus.cc:2454] T dc96c465fe554a94aa599200d717a3c2 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5af98ef61b54ae0b33faeb78af96dec in term 1.
I20240501 07:59:30.955807 1320 consensus_queue.cc:237] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:30.957820 439 catalog_manager.cc:5484] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec reported cstate change: term changed from 0 to 1, leader changed from <none> to f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133). New cstate: current_term: 1 leader_uuid: "f5af98ef61b54ae0b33faeb78af96dec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:30.970438 1493 raft_consensus.cc:482] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:30.970577 1493 raft_consensus.cc:504] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:30.971040 1493 leader_election.cc:288] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c45c306768564fb2bd741c535d334429 (127.0.99.134:37067), 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765)
I20240501 07:59:30.971491 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5240af8639074b7b92c38021d56b3482" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429" is_pre_election: true
I20240501 07:59:30.971509 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5240af8639074b7b92c38021d56b3482" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725" is_pre_election: true
I20240501 07:59:30.971632 715 raft_consensus.cc:2454] T 5240af8639074b7b92c38021d56b3482 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:30.971635 1243 raft_consensus.cc:2454] T 5240af8639074b7b92c38021d56b3482 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:30.971949 913 leader_election.cc:302] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, 3ee011df676c43f7beaeace0f8c60725; no voters:
I20240501 07:59:30.972121 1493 raft_consensus.cc:2781] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:30.972198 1493 raft_consensus.cc:482] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:30.972250 1493 raft_consensus.cc:3037] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.046253 1493 raft_consensus.cc:504] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:31.046751 1493 leader_election.cc:288] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Requested vote from peers c45c306768564fb2bd741c535d334429 (127.0.99.134:37067), 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765)
I20240501 07:59:31.047262 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5240af8639074b7b92c38021d56b3482" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429"
I20240501 07:59:31.047279 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5240af8639074b7b92c38021d56b3482" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725"
I20240501 07:59:31.047394 1243 raft_consensus.cc:3037] T 5240af8639074b7b92c38021d56b3482 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.047448 715 raft_consensus.cc:3037] T 5240af8639074b7b92c38021d56b3482 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.048408 1243 raft_consensus.cc:2454] T 5240af8639074b7b92c38021d56b3482 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:31.048717 715 raft_consensus.cc:2454] T 5240af8639074b7b92c38021d56b3482 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:31.048885 912 leader_election.cc:302] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:31.049085 1493 raft_consensus.cc:2781] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:31.049280 1493 raft_consensus.cc:686] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [term 1 LEADER]: Becoming Leader. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Running, Role: LEADER
I20240501 07:59:31.049429 1493 consensus_queue.cc:237] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:31.051044 439 catalog_manager.cc:5484] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1526c5008f714bcfa69e9bacb51a662c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: HEALTHY } } }
W20240501 07:59:31.226235 1607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:31.226514 1607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:31.229874 1607 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 07:59:31.323215 1346 consensus_queue.cc:1035] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:31.332995 1320 consensus_queue.cc:1035] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:31.335845 1346 consensus_queue.cc:1035] T 549c8b9eef0440979653e95da1523966 P 3ee011df676c43f7beaeace0f8c60725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:31.339754 1603 consensus_queue.cc:1035] T dc96c465fe554a94aa599200d717a3c2 P f5af98ef61b54ae0b33faeb78af96dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:31.341789 1242 raft_consensus.cc:1261] T 5240af8639074b7b92c38021d56b3482 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Refusing update from remote peer 1526c5008f714bcfa69e9bacb51a662c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:31.341965 715 raft_consensus.cc:1261] T 5240af8639074b7b92c38021d56b3482 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Refusing update from remote peer 1526c5008f714bcfa69e9bacb51a662c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:31.342423 1493 consensus_queue.cc:1035] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:31.342729 1493 consensus_queue.cc:1035] T 5240af8639074b7b92c38021d56b3482 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 132.811 ms
time per row: 0.0664054 ms
I20240501 07:59:31.411430 439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36176:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:31.417446 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 12eff9f6cf9545d1bbd326bcb0e84b22 (DEFAULT_TABLE table=rebalance_test_table_4 [id=36c91399ebe14985898958c6ce198df2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.417503 563 tablet_service.cc:1467] Processing CreateTablet for tablet 12eff9f6cf9545d1bbd326bcb0e84b22 (DEFAULT_TABLE table=rebalance_test_table_4 [id=36c91399ebe14985898958c6ce198df2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.417778 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12eff9f6cf9545d1bbd326bcb0e84b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.417824 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12eff9f6cf9545d1bbd326bcb0e84b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.419797 1676 tablet_bootstrap.cc:492] T 12eff9f6cf9545d1bbd326bcb0e84b22 P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:31.420096 1677 tablet_bootstrap.cc:492] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:31.420286 826 tablet_service.cc:1467] Processing CreateTablet for tablet 12eff9f6cf9545d1bbd326bcb0e84b22 (DEFAULT_TABLE table=rebalance_test_table_4 [id=36c91399ebe14985898958c6ce198df2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.420564 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12eff9f6cf9545d1bbd326bcb0e84b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.420686 1676 tablet_bootstrap.cc:654] T 12eff9f6cf9545d1bbd326bcb0e84b22 P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.420717 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 857f176f0500475894550b58d43e422b (DEFAULT_TABLE table=rebalance_test_table_4 [id=36c91399ebe14985898958c6ce198df2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.420972 1677 tablet_bootstrap.cc:654] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.421012 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857f176f0500475894550b58d43e422b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.422137 1677 tablet_bootstrap.cc:492] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:31.422226 1677 ts_tablet_manager.cc:1367] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:31.422946 827 tablet_service.cc:1467] Processing CreateTablet for tablet c9138f65fa2a4d76ba9ee66119ade3ef (DEFAULT_TABLE table=rebalance_test_table_4 [id=36c91399ebe14985898958c6ce198df2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.423214 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9138f65fa2a4d76ba9ee66119ade3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.423352 695 tablet_service.cc:1467] Processing CreateTablet for tablet 857f176f0500475894550b58d43e422b (DEFAULT_TABLE table=rebalance_test_table_4 [id=36c91399ebe14985898958c6ce198df2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.423408 1676 tablet_bootstrap.cc:492] T 12eff9f6cf9545d1bbd326bcb0e84b22 P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:31.423496 1676 ts_tablet_manager.cc:1367] T 12eff9f6cf9545d1bbd326bcb0e84b22 P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:31.423620 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857f176f0500475894550b58d43e422b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.423800 959 tablet_service.cc:1467] Processing CreateTablet for tablet 857f176f0500475894550b58d43e422b (DEFAULT_TABLE table=rebalance_test_table_4 [id=36c91399ebe14985898958c6ce198df2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.423974 1676 raft_consensus.cc:348] T 12eff9f6cf9545d1bbd326bcb0e84b22 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:31.424070 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857f176f0500475894550b58d43e422b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.424101 1676 raft_consensus.cc:374] T 12eff9f6cf9545d1bbd326bcb0e84b22 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.424134 1676 raft_consensus.cc:729] T 12eff9f6cf9545d1bbd326bcb0e84b22 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.424261 1676 consensus_queue.cc:260] T 12eff9f6cf9545d1bbd326bcb0e84b22 P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:31.424531 1676 ts_tablet_manager.cc:1398] T 12eff9f6cf9545d1bbd326bcb0e84b22 P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:31.425976 1682 tablet_bootstrap.cc:492] T 857f176f0500475894550b58d43e422b P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:31.426254 1677 raft_consensus.cc:348] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:31.426450 1677 raft_consensus.cc:374] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.426481 1090 tablet_service.cc:1467] Processing CreateTablet for tablet c9138f65fa2a4d76ba9ee66119ade3ef (DEFAULT_TABLE table=rebalance_test_table_4 [id=36c91399ebe14985898958c6ce198df2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.426537 1677 raft_consensus.cc:729] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.426800 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9138f65fa2a4d76ba9ee66119ade3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.426978 1683 tablet_bootstrap.cc:492] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:31.427004 1682 tablet_bootstrap.cc:654] T 857f176f0500475894550b58d43e422b P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.428309 1683 tablet_bootstrap.cc:654] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.428364 563 tablet_service.cc:1467] Processing CreateTablet for tablet c9138f65fa2a4d76ba9ee66119ade3ef (DEFAULT_TABLE table=rebalance_test_table_4 [id=36c91399ebe14985898958c6ce198df2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.428570 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9138f65fa2a4d76ba9ee66119ade3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.426803 1677 consensus_queue.cc:260] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:31.429456 1683 tablet_bootstrap.cc:492] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:31.429589 1677 ts_tablet_manager.cc:1398] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 07:59:31.429620 1683 ts_tablet_manager.cc:1367] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:31.429811 1677 tablet_bootstrap.cc:492] T c9138f65fa2a4d76ba9ee66119ade3ef P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:31.430167 1683 raft_consensus.cc:348] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:31.430353 1683 raft_consensus.cc:374] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.430457 1683 raft_consensus.cc:729] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.430702 1683 consensus_queue.cc:260] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:31.431123 1683 ts_tablet_manager.cc:1398] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:31.431192 1677 tablet_bootstrap.cc:654] T c9138f65fa2a4d76ba9ee66119ade3ef P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.431953 1680 tablet_bootstrap.cc:492] T 12eff9f6cf9545d1bbd326bcb0e84b22 P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:31.432029 1682 tablet_bootstrap.cc:492] T 857f176f0500475894550b58d43e422b P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:31.432132 1682 ts_tablet_manager.cc:1367] T 857f176f0500475894550b58d43e422b P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:31.432405 1677 tablet_bootstrap.cc:492] T c9138f65fa2a4d76ba9ee66119ade3ef P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:31.432538 1677 ts_tablet_manager.cc:1367] T c9138f65fa2a4d76ba9ee66119ade3ef P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:31.432633 1682 raft_consensus.cc:348] T 857f176f0500475894550b58d43e422b P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:31.432821 1682 raft_consensus.cc:374] T 857f176f0500475894550b58d43e422b P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.432909 1682 raft_consensus.cc:729] T 857f176f0500475894550b58d43e422b P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.433055 1677 raft_consensus.cc:348] T c9138f65fa2a4d76ba9ee66119ade3ef P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:31.433102 1682 consensus_queue.cc:260] T 857f176f0500475894550b58d43e422b P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:31.433257 1677 raft_consensus.cc:374] T c9138f65fa2a4d76ba9ee66119ade3ef P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.433332 1677 raft_consensus.cc:729] T c9138f65fa2a4d76ba9ee66119ade3ef P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.433431 1682 ts_tablet_manager.cc:1398] T 857f176f0500475894550b58d43e422b P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:31.433524 1677 consensus_queue.cc:260] T c9138f65fa2a4d76ba9ee66119ade3ef P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:31.433864 1677 ts_tablet_manager.cc:1398] T c9138f65fa2a4d76ba9ee66119ade3ef P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:31.434720 1676 tablet_bootstrap.cc:492] T c9138f65fa2a4d76ba9ee66119ade3ef P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:31.435773 1680 tablet_bootstrap.cc:654] T 12eff9f6cf9545d1bbd326bcb0e84b22 P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.435798 1676 tablet_bootstrap.cc:654] T c9138f65fa2a4d76ba9ee66119ade3ef P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.437330 1676 tablet_bootstrap.cc:492] T c9138f65fa2a4d76ba9ee66119ade3ef P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:31.437357 1680 tablet_bootstrap.cc:492] T 12eff9f6cf9545d1bbd326bcb0e84b22 P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:31.437429 1676 ts_tablet_manager.cc:1367] T c9138f65fa2a4d76ba9ee66119ade3ef P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:31.437429 1680 ts_tablet_manager.cc:1367] T 12eff9f6cf9545d1bbd326bcb0e84b22 P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:31.437873 1680 raft_consensus.cc:348] T 12eff9f6cf9545d1bbd326bcb0e84b22 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:31.437867 1676 raft_consensus.cc:348] T c9138f65fa2a4d76ba9ee66119ade3ef P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:31.438014 1676 raft_consensus.cc:374] T c9138f65fa2a4d76ba9ee66119ade3ef P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.438052 1680 raft_consensus.cc:374] T 12eff9f6cf9545d1bbd326bcb0e84b22 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.438059 1676 raft_consensus.cc:729] T c9138f65fa2a4d76ba9ee66119ade3ef P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.438095 1680 raft_consensus.cc:729] T 12eff9f6cf9545d1bbd326bcb0e84b22 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.438211 1676 consensus_queue.cc:260] T c9138f65fa2a4d76ba9ee66119ade3ef P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:31.438270 1680 consensus_queue.cc:260] T 12eff9f6cf9545d1bbd326bcb0e84b22 P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:31.438517 1676 ts_tablet_manager.cc:1398] T c9138f65fa2a4d76ba9ee66119ade3ef P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:31.438542 1680 ts_tablet_manager.cc:1398] T 12eff9f6cf9545d1bbd326bcb0e84b22 P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:31.438714 1680 tablet_bootstrap.cc:492] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:31.438892 1681 tablet_bootstrap.cc:492] T 857f176f0500475894550b58d43e422b P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:31.439788 1680 tablet_bootstrap.cc:654] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.439975 1681 tablet_bootstrap.cc:654] T 857f176f0500475894550b58d43e422b P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.441291 1681 tablet_bootstrap.cc:492] T 857f176f0500475894550b58d43e422b P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:31.441398 1681 ts_tablet_manager.cc:1367] T 857f176f0500475894550b58d43e422b P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:31.441918 1681 raft_consensus.cc:348] T 857f176f0500475894550b58d43e422b P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:31.442066 1681 raft_consensus.cc:374] T 857f176f0500475894550b58d43e422b P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.442108 1681 raft_consensus.cc:729] T 857f176f0500475894550b58d43e422b P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.442252 1681 consensus_queue.cc:260] T 857f176f0500475894550b58d43e422b P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:31.442286 1680 tablet_bootstrap.cc:492] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:31.442368 1680 ts_tablet_manager.cc:1367] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:59:31.442530 1681 ts_tablet_manager.cc:1398] T 857f176f0500475894550b58d43e422b P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:31.442790 1680 raft_consensus.cc:348] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:31.442885 1680 raft_consensus.cc:374] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.442907 1680 raft_consensus.cc:729] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.443024 1680 consensus_queue.cc:260] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:31.443306 1680 ts_tablet_manager.cc:1398] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:31.457158 1559 raft_consensus.cc:482] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:31.457293 1559 raft_consensus.cc:504] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:31.457731 1559 leader_election.cc:288] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:31.458156 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9138f65fa2a4d76ba9ee66119ade3ef" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747" is_pre_election: true
I20240501 07:59:31.458182 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9138f65fa2a4d76ba9ee66119ade3ef" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec" is_pre_election: true
I20240501 07:59:31.458320 1106 raft_consensus.cc:2454] T c9138f65fa2a4d76ba9ee66119ade3ef P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:31.458319 583 raft_consensus.cc:2454] T c9138f65fa2a4d76ba9ee66119ade3ef P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:31.458639 780 leader_election.cc:302] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:31.458818 1559 raft_consensus.cc:2781] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:31.458875 1559 raft_consensus.cc:482] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:31.458925 1559 raft_consensus.cc:3037] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.492266 1559 raft_consensus.cc:504] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:31.492815 1559 leader_election.cc:288] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Requested vote from peers b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:31.493345 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9138f65fa2a4d76ba9ee66119ade3ef" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747"
I20240501 07:59:31.493374 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9138f65fa2a4d76ba9ee66119ade3ef" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec"
I20240501 07:59:31.493485 583 raft_consensus.cc:3037] T c9138f65fa2a4d76ba9ee66119ade3ef P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.493547 1106 raft_consensus.cc:3037] T c9138f65fa2a4d76ba9ee66119ade3ef P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.494429 583 raft_consensus.cc:2454] T c9138f65fa2a4d76ba9ee66119ade3ef P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:31.494788 1106 raft_consensus.cc:2454] T c9138f65fa2a4d76ba9ee66119ade3ef P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:31.494833 781 leader_election.cc:302] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, b9644aeae9de4deba0c3a2b85df04747; no voters:
I20240501 07:59:31.495074 1559 raft_consensus.cc:2781] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:31.495292 1559 raft_consensus.cc:686] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee66e185465431389043d933cb51308, State: Running, Role: LEADER
I20240501 07:59:31.495460 1559 consensus_queue.cc:237] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:31.497311 439 catalog_manager.cc:5484] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee66e185465431389043d933cb51308 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8ee66e185465431389043d933cb51308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:31.712483 1493 raft_consensus.cc:482] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:31.712623 1493 raft_consensus.cc:504] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:31.713064 1493 leader_election.cc:288] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:31.713440 1320 raft_consensus.cc:482] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:31.713558 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857f176f0500475894550b58d43e422b" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725" is_pre_election: true
I20240501 07:59:31.713562 1320 raft_consensus.cc:504] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:31.713558 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857f176f0500475894550b58d43e422b" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429" is_pre_election: true
I20240501 07:59:31.713708 715 raft_consensus.cc:2454] T 857f176f0500475894550b58d43e422b P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:31.713732 1242 raft_consensus.cc:2454] T 857f176f0500475894550b58d43e422b P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:31.713997 1320 leader_election.cc:288] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609), 8ee66e185465431389043d933cb51308 (127.0.99.131:37541)
I20240501 07:59:31.714051 912 leader_election.cc:302] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:31.714231 1493 raft_consensus.cc:2781] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:31.714289 1493 raft_consensus.cc:482] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:31.714329 1493 raft_consensus.cc:3037] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.714463 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12eff9f6cf9545d1bbd326bcb0e84b22" candidate_uuid: "f5af98ef61b54ae0b33faeb78af96dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308" is_pre_election: true
I20240501 07:59:31.714500 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12eff9f6cf9545d1bbd326bcb0e84b22" candidate_uuid: "f5af98ef61b54ae0b33faeb78af96dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747" is_pre_election: true
I20240501 07:59:31.714610 846 raft_consensus.cc:2454] T 12eff9f6cf9545d1bbd326bcb0e84b22 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5af98ef61b54ae0b33faeb78af96dec in term 0.
I20240501 07:59:31.714625 583 raft_consensus.cc:2454] T 12eff9f6cf9545d1bbd326bcb0e84b22 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5af98ef61b54ae0b33faeb78af96dec in term 0.
I20240501 07:59:31.714916 1046 leader_election.cc:302] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:31.715075 1320 raft_consensus.cc:2781] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:31.715134 1320 raft_consensus.cc:482] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:31.715181 1320 raft_consensus.cc:3037] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.732662 1493 raft_consensus.cc:504] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:31.732702 1320 raft_consensus.cc:504] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:31.733304 1493 leader_election.cc:288] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Requested vote from peers 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:31.733359 1320 leader_election.cc:288] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [CANDIDATE]: Term 1 election: Requested vote from peers b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609), 8ee66e185465431389043d933cb51308 (127.0.99.131:37541)
I20240501 07:59:31.734053 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857f176f0500475894550b58d43e422b" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429"
I20240501 07:59:31.734068 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12eff9f6cf9545d1bbd326bcb0e84b22" candidate_uuid: "f5af98ef61b54ae0b33faeb78af96dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308"
I20240501 07:59:31.734198 1242 raft_consensus.cc:3037] T 857f176f0500475894550b58d43e422b P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.734210 847 raft_consensus.cc:3037] T 12eff9f6cf9545d1bbd326bcb0e84b22 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.735147 1242 raft_consensus.cc:2454] T 857f176f0500475894550b58d43e422b P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:31.735399 847 raft_consensus.cc:2454] T 12eff9f6cf9545d1bbd326bcb0e84b22 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5af98ef61b54ae0b33faeb78af96dec in term 1.
I20240501 07:59:31.735527 912 leader_election.cc:302] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:31.735728 1493 raft_consensus.cc:2781] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:31.735703 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857f176f0500475894550b58d43e422b" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725"
I20240501 07:59:31.735839 715 raft_consensus.cc:3037] T 857f176f0500475894550b58d43e422b P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.735846 1046 leader_election.cc:302] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:31.736125 1493 raft_consensus.cc:686] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [term 1 LEADER]: Becoming Leader. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Running, Role: LEADER
I20240501 07:59:31.736245 1320 raft_consensus.cc:2781] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:31.736296 1493 consensus_queue.cc:237] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:31.736501 1320 raft_consensus.cc:686] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [term 1 LEADER]: Becoming Leader. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Running, Role: LEADER
I20240501 07:59:31.736655 1320 consensus_queue.cc:237] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:31.737062 715 raft_consensus.cc:2454] T 857f176f0500475894550b58d43e422b P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:31.738992 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12eff9f6cf9545d1bbd326bcb0e84b22" candidate_uuid: "f5af98ef61b54ae0b33faeb78af96dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747"
I20240501 07:59:31.739115 583 raft_consensus.cc:3037] T 12eff9f6cf9545d1bbd326bcb0e84b22 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.739302 439 catalog_manager.cc:5484] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1526c5008f714bcfa69e9bacb51a662c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:31.739301 437 catalog_manager.cc:5484] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec reported cstate change: term changed from 0 to 1, leader changed from <none> to f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133). New cstate: current_term: 1 leader_uuid: "f5af98ef61b54ae0b33faeb78af96dec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:31.804390 583 raft_consensus.cc:2454] T 12eff9f6cf9545d1bbd326bcb0e84b22 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5af98ef61b54ae0b33faeb78af96dec in term 1.
I20240501 07:59:31.899416 1559 consensus_queue.cc:1035] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:31.905077 1691 consensus_queue.cc:1035] T c9138f65fa2a4d76ba9ee66119ade3ef P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:59:31.918291 1696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:31.918563 1696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:31.921880 1696 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 07:59:32.019495 847 raft_consensus.cc:1261] T 12eff9f6cf9545d1bbd326bcb0e84b22 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Refusing update from remote peer f5af98ef61b54ae0b33faeb78af96dec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.019949 583 raft_consensus.cc:1261] T 12eff9f6cf9545d1bbd326bcb0e84b22 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Refusing update from remote peer f5af98ef61b54ae0b33faeb78af96dec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.020478 1320 consensus_queue.cc:1035] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.020812 1603 consensus_queue.cc:1035] T 12eff9f6cf9545d1bbd326bcb0e84b22 P f5af98ef61b54ae0b33faeb78af96dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.031332 1242 raft_consensus.cc:1261] T 857f176f0500475894550b58d43e422b P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Refusing update from remote peer 1526c5008f714bcfa69e9bacb51a662c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.031756 1493 consensus_queue.cc:1035] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.033759 715 raft_consensus.cc:1261] T 857f176f0500475894550b58d43e422b P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Refusing update from remote peer 1526c5008f714bcfa69e9bacb51a662c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.034214 1493 consensus_queue.cc:1035] T 857f176f0500475894550b58d43e422b P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 132.057 ms
time per row: 0.0660286 ms
I20240501 07:59:32.103988 440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36176:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:32.110289 959 tablet_service.cc:1467] Processing CreateTablet for tablet 142a1237964a40d1b7e5ce852633be44 (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e619c883f04955b113a6c3daf2aceb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.110293 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 142a1237964a40d1b7e5ce852633be44 (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e619c883f04955b113a6c3daf2aceb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.110636 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142a1237964a40d1b7e5ce852633be44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.110636 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142a1237964a40d1b7e5ce852633be44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.110625 694 tablet_service.cc:1467] Processing CreateTablet for tablet c99a78b4f41244a9a95dd627ef356142 (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e619c883f04955b113a6c3daf2aceb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.110936 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99a78b4f41244a9a95dd627ef356142. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.112931 1760 tablet_bootstrap.cc:492] T c99a78b4f41244a9a95dd627ef356142 P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:32.113262 695 tablet_service.cc:1467] Processing CreateTablet for tablet 142a1237964a40d1b7e5ce852633be44 (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e619c883f04955b113a6c3daf2aceb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.113557 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142a1237964a40d1b7e5ce852633be44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.113814 1761 tablet_bootstrap.cc:492] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:32.114084 1760 tablet_bootstrap.cc:654] T c99a78b4f41244a9a95dd627ef356142 P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.114756 1761 tablet_bootstrap.cc:654] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.115187 1760 tablet_bootstrap.cc:492] T c99a78b4f41244a9a95dd627ef356142 P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:32.115271 1760 ts_tablet_manager.cc:1367] T c99a78b4f41244a9a95dd627ef356142 P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:32.115460 1762 tablet_bootstrap.cc:492] T 142a1237964a40d1b7e5ce852633be44 P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:32.115677 1760 raft_consensus.cc:348] T c99a78b4f41244a9a95dd627ef356142 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:32.115777 1760 raft_consensus.cc:374] T c99a78b4f41244a9a95dd627ef356142 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.115808 1760 raft_consensus.cc:729] T c99a78b4f41244a9a95dd627ef356142 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.115931 1760 consensus_queue.cc:260] T c99a78b4f41244a9a95dd627ef356142 P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:32.116194 1760 ts_tablet_manager.cc:1398] T c99a78b4f41244a9a95dd627ef356142 P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:32.116361 1760 tablet_bootstrap.cc:492] T 142a1237964a40d1b7e5ce852633be44 P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:32.116449 827 tablet_service.cc:1467] Processing CreateTablet for tablet 27db9a40cf144c4a9018e906ee72e54f (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e619c883f04955b113a6c3daf2aceb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.116616 1762 tablet_bootstrap.cc:654] T 142a1237964a40d1b7e5ce852633be44 P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.116714 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27db9a40cf144c4a9018e906ee72e54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.117259 1760 tablet_bootstrap.cc:654] T 142a1237964a40d1b7e5ce852633be44 P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.117581 1761 tablet_bootstrap.cc:492] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:32.117663 1761 ts_tablet_manager.cc:1367] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 07:59:32.117730 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 27db9a40cf144c4a9018e906ee72e54f (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e619c883f04955b113a6c3daf2aceb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.118621 1760 tablet_bootstrap.cc:492] T 142a1237964a40d1b7e5ce852633be44 P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:32.118713 1760 ts_tablet_manager.cc:1367] T 142a1237964a40d1b7e5ce852633be44 P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:32.118672 563 tablet_service.cc:1467] Processing CreateTablet for tablet 27db9a40cf144c4a9018e906ee72e54f (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e619c883f04955b113a6c3daf2aceb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.118974 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27db9a40cf144c4a9018e906ee72e54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.118116 1761 raft_consensus.cc:348] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:32.119179 1761 raft_consensus.cc:374] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.119218 1761 raft_consensus.cc:729] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.119164 1760 raft_consensus.cc:348] T 142a1237964a40d1b7e5ce852633be44 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:32.119266 1760 raft_consensus.cc:374] T 142a1237964a40d1b7e5ce852633be44 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.119297 1760 raft_consensus.cc:729] T 142a1237964a40d1b7e5ce852633be44 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.119388 1761 consensus_queue.cc:260] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:32.119419 1760 consensus_queue.cc:260] T 142a1237964a40d1b7e5ce852633be44 P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:32.119665 1760 ts_tablet_manager.cc:1398] T 142a1237964a40d1b7e5ce852633be44 P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:32.120067 1761 ts_tablet_manager.cc:1398] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:32.120298 1767 tablet_bootstrap.cc:492] T 27db9a40cf144c4a9018e906ee72e54f P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:32.121011 1768 tablet_bootstrap.cc:492] T 27db9a40cf144c4a9018e906ee72e54f P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:32.121029 1767 tablet_bootstrap.cc:654] T 27db9a40cf144c4a9018e906ee72e54f P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.122117 1767 tablet_bootstrap.cc:492] T 27db9a40cf144c4a9018e906ee72e54f P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:32.122213 1767 ts_tablet_manager.cc:1367] T 27db9a40cf144c4a9018e906ee72e54f P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:32.122637 1768 tablet_bootstrap.cc:654] T 27db9a40cf144c4a9018e906ee72e54f P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.122586 1767 raft_consensus.cc:348] T 27db9a40cf144c4a9018e906ee72e54f P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } }
I20240501 07:59:32.122687 1767 raft_consensus.cc:374] T 27db9a40cf144c4a9018e906ee72e54f P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.122709 1767 raft_consensus.cc:729] T 27db9a40cf144c4a9018e906ee72e54f P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.122825 1767 consensus_queue.cc:260] T 27db9a40cf144c4a9018e906ee72e54f P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } }
I20240501 07:59:32.123075 1767 ts_tablet_manager.cc:1398] T 27db9a40cf144c4a9018e906ee72e54f P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:32.124140 1768 tablet_bootstrap.cc:492] T 27db9a40cf144c4a9018e906ee72e54f P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:32.124243 1768 ts_tablet_manager.cc:1367] T 27db9a40cf144c4a9018e906ee72e54f P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:32.124172 1222 tablet_service.cc:1467] Processing CreateTablet for tablet c99a78b4f41244a9a95dd627ef356142 (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e619c883f04955b113a6c3daf2aceb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.124455 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99a78b4f41244a9a95dd627ef356142. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.124990 1768 raft_consensus.cc:348] T 27db9a40cf144c4a9018e906ee72e54f P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } }
I20240501 07:59:32.125180 1768 raft_consensus.cc:374] T 27db9a40cf144c4a9018e906ee72e54f P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.125258 1768 raft_consensus.cc:729] T 27db9a40cf144c4a9018e906ee72e54f P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.125447 1768 consensus_queue.cc:260] T 27db9a40cf144c4a9018e906ee72e54f P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } }
I20240501 07:59:32.125826 1768 ts_tablet_manager.cc:1398] T 27db9a40cf144c4a9018e906ee72e54f P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:32.126091 959 tablet_service.cc:1467] Processing CreateTablet for tablet c99a78b4f41244a9a95dd627ef356142 (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e619c883f04955b113a6c3daf2aceb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.126390 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99a78b4f41244a9a95dd627ef356142. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.128284 1761 tablet_bootstrap.cc:492] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:32.129417 1761 tablet_bootstrap.cc:654] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.129914 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27db9a40cf144c4a9018e906ee72e54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.130896 1761 tablet_bootstrap.cc:492] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:32.130977 1761 ts_tablet_manager.cc:1367] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:59:32.131450 1761 raft_consensus.cc:348] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:32.131549 1761 raft_consensus.cc:374] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.131577 1761 raft_consensus.cc:729] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.131691 1761 consensus_queue.cc:260] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:32.132485 1772 tablet_bootstrap.cc:492] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:32.132539 1761 ts_tablet_manager.cc:1398] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:32.133586 1762 tablet_bootstrap.cc:492] T 142a1237964a40d1b7e5ce852633be44 P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:32.133599 1772 tablet_bootstrap.cc:654] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.133697 1762 ts_tablet_manager.cc:1367] T 142a1237964a40d1b7e5ce852633be44 P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 07:59:32.134238 1762 raft_consensus.cc:348] T 142a1237964a40d1b7e5ce852633be44 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:32.134375 1762 raft_consensus.cc:374] T 142a1237964a40d1b7e5ce852633be44 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.134413 1762 raft_consensus.cc:729] T 142a1237964a40d1b7e5ce852633be44 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.134549 1762 consensus_queue.cc:260] T 142a1237964a40d1b7e5ce852633be44 P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:32.134846 1762 ts_tablet_manager.cc:1398] T 142a1237964a40d1b7e5ce852633be44 P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:32.134950 1772 tablet_bootstrap.cc:492] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:32.135061 1772 ts_tablet_manager.cc:1367] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:32.135149 1762 tablet_bootstrap.cc:492] T c99a78b4f41244a9a95dd627ef356142 P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:32.136050 1762 tablet_bootstrap.cc:654] T c99a78b4f41244a9a95dd627ef356142 P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.135499 1772 raft_consensus.cc:348] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } }
I20240501 07:59:32.136538 1772 raft_consensus.cc:374] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.136577 1772 raft_consensus.cc:729] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.136740 1772 consensus_queue.cc:260] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } }
I20240501 07:59:32.137863 1762 tablet_bootstrap.cc:492] T c99a78b4f41244a9a95dd627ef356142 P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:32.138024 1762 ts_tablet_manager.cc:1367] T c99a78b4f41244a9a95dd627ef356142 P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:32.138571 1762 raft_consensus.cc:348] T c99a78b4f41244a9a95dd627ef356142 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:32.138682 1772 ts_tablet_manager.cc:1398] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:32.138769 1762 raft_consensus.cc:374] T c99a78b4f41244a9a95dd627ef356142 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.138849 1762 raft_consensus.cc:729] T c99a78b4f41244a9a95dd627ef356142 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.138978 1762 consensus_queue.cc:260] T c99a78b4f41244a9a95dd627ef356142 P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:32.139348 1762 ts_tablet_manager.cc:1398] T c99a78b4f41244a9a95dd627ef356142 P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:32.143168 1605 raft_consensus.cc:482] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:32.143317 1605 raft_consensus.cc:504] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:32.143829 1605 leader_election.cc:288] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:32.144189 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99a78b4f41244a9a95dd627ef356142" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429" is_pre_election: true
I20240501 07:59:32.144330 1242 raft_consensus.cc:2454] T c99a78b4f41244a9a95dd627ef356142 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:32.144629 912 leader_election.cc:302] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:32.144850 1605 raft_consensus.cc:2781] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:32.144937 1605 raft_consensus.cc:482] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:32.145001 1605 raft_consensus.cc:3037] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.146090 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99a78b4f41244a9a95dd627ef356142" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725" is_pre_election: true
I20240501 07:59:32.146224 715 raft_consensus.cc:2454] T c99a78b4f41244a9a95dd627ef356142 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:32.180171 1493 raft_consensus.cc:482] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:32.180334 1493 raft_consensus.cc:504] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:32.180744 1493 leader_election.cc:288] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c45c306768564fb2bd741c535d334429 (127.0.99.134:37067), 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765)
I20240501 07:59:32.181234 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142a1237964a40d1b7e5ce852633be44" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725" is_pre_election: true
I20240501 07:59:32.181252 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142a1237964a40d1b7e5ce852633be44" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429" is_pre_election: true
I20240501 07:59:32.181397 1242 raft_consensus.cc:2454] T 142a1237964a40d1b7e5ce852633be44 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:32.181397 715 raft_consensus.cc:2454] T 142a1237964a40d1b7e5ce852633be44 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:32.181715 913 leader_election.cc:302] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, 3ee011df676c43f7beaeace0f8c60725; no voters:
I20240501 07:59:32.181915 1493 raft_consensus.cc:2781] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:32.182014 1493 raft_consensus.cc:482] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:32.182067 1493 raft_consensus.cc:3037] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.196607 1746 raft_consensus.cc:482] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:32.196787 1746 raft_consensus.cc:504] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } }
I20240501 07:59:32.197250 1746 leader_election.cc:288] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee66e185465431389043d933cb51308 (127.0.99.131:37541), b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609)
I20240501 07:59:32.197732 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27db9a40cf144c4a9018e906ee72e54f" candidate_uuid: "f5af98ef61b54ae0b33faeb78af96dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308" is_pre_election: true
I20240501 07:59:32.197793 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27db9a40cf144c4a9018e906ee72e54f" candidate_uuid: "f5af98ef61b54ae0b33faeb78af96dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747" is_pre_election: true
I20240501 07:59:32.197886 847 raft_consensus.cc:2454] T 27db9a40cf144c4a9018e906ee72e54f P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5af98ef61b54ae0b33faeb78af96dec in term 0.
I20240501 07:59:32.197955 582 raft_consensus.cc:2454] T 27db9a40cf144c4a9018e906ee72e54f P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5af98ef61b54ae0b33faeb78af96dec in term 0.
I20240501 07:59:32.198315 1045 leader_election.cc:302] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9644aeae9de4deba0c3a2b85df04747, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:32.198501 1746 raft_consensus.cc:2781] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:32.198563 1746 raft_consensus.cc:482] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:32.198592 1746 raft_consensus.cc:3037] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.243818 1746 raft_consensus.cc:504] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } }
I20240501 07:59:32.243818 1605 raft_consensus.cc:504] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:32.244344 1746 leader_election.cc:288] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [CANDIDATE]: Term 1 election: Requested vote from peers 8ee66e185465431389043d933cb51308 (127.0.99.131:37541), b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609)
I20240501 07:59:32.244386 1605 leader_election.cc:288] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Requested vote from peers 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:32.244657 1493 raft_consensus.cc:504] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:32.245023 1493 leader_election.cc:288] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Requested vote from peers c45c306768564fb2bd741c535d334429 (127.0.99.134:37067), 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765)
I20240501 07:59:32.245067 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27db9a40cf144c4a9018e906ee72e54f" candidate_uuid: "f5af98ef61b54ae0b33faeb78af96dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308"
I20240501 07:59:32.245205 847 raft_consensus.cc:3037] T 27db9a40cf144c4a9018e906ee72e54f P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.245533 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142a1237964a40d1b7e5ce852633be44" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725"
I20240501 07:59:32.245652 714 raft_consensus.cc:3037] T 142a1237964a40d1b7e5ce852633be44 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.245604 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27db9a40cf144c4a9018e906ee72e54f" candidate_uuid: "f5af98ef61b54ae0b33faeb78af96dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747"
I20240501 07:59:32.245739 582 raft_consensus.cc:3037] T 27db9a40cf144c4a9018e906ee72e54f P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.246299 847 raft_consensus.cc:2454] T 27db9a40cf144c4a9018e906ee72e54f P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5af98ef61b54ae0b33faeb78af96dec in term 1.
I20240501 07:59:32.246675 1046 leader_election.cc:302] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:32.246732 714 raft_consensus.cc:2454] T 142a1237964a40d1b7e5ce852633be44 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:32.246884 1746 raft_consensus.cc:2781] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:32.246999 1746 raft_consensus.cc:686] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [term 1 LEADER]: Becoming Leader. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Running, Role: LEADER
I20240501 07:59:32.247030 582 raft_consensus.cc:2454] T 27db9a40cf144c4a9018e906ee72e54f P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5af98ef61b54ae0b33faeb78af96dec in term 1.
I20240501 07:59:32.247061 913 leader_election.cc:302] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, 3ee011df676c43f7beaeace0f8c60725; no voters:
I20240501 07:59:32.247141 1746 consensus_queue.cc:237] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } }
I20240501 07:59:32.247668 1493 raft_consensus.cc:2781] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:32.247681 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142a1237964a40d1b7e5ce852633be44" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429"
I20240501 07:59:32.247758 1493 raft_consensus.cc:686] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [term 1 LEADER]: Becoming Leader. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Running, Role: LEADER
I20240501 07:59:32.247810 1243 raft_consensus.cc:3037] T 142a1237964a40d1b7e5ce852633be44 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.247797 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99a78b4f41244a9a95dd627ef356142" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725"
I20240501 07:59:32.247911 715 raft_consensus.cc:3037] T c99a78b4f41244a9a95dd627ef356142 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.247884 1493 consensus_queue.cc:237] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:32.248848 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99a78b4f41244a9a95dd627ef356142" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429"
I20240501 07:59:32.248958 1242 raft_consensus.cc:3037] T c99a78b4f41244a9a95dd627ef356142 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.250485 440 catalog_manager.cc:5484] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1526c5008f714bcfa69e9bacb51a662c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:32.250983 441 catalog_manager.cc:5484] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec reported cstate change: term changed from 0 to 1, leader changed from <none> to f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133). New cstate: current_term: 1 leader_uuid: "f5af98ef61b54ae0b33faeb78af96dec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:32.341234 1243 raft_consensus.cc:2454] T 142a1237964a40d1b7e5ce852633be44 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:32.341230 715 raft_consensus.cc:2454] T c99a78b4f41244a9a95dd627ef356142 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:32.341737 913 leader_election.cc:302] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, 3ee011df676c43f7beaeace0f8c60725; no voters:
I20240501 07:59:32.341961 1493 raft_consensus.cc:2781] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:32.342108 1493 raft_consensus.cc:686] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [term 1 LEADER]: Becoming Leader. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Running, Role: LEADER
I20240501 07:59:32.342446 1493 consensus_queue.cc:237] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:32.344317 441 catalog_manager.cc:5484] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1526c5008f714bcfa69e9bacb51a662c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:32.373531 1242 raft_consensus.cc:2454] T c99a78b4f41244a9a95dd627ef356142 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
W20240501 07:59:32.538585 1780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:32.538887 1780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:32.542614 1780 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 07:59:32.632690 1242 raft_consensus.cc:1261] T c99a78b4f41244a9a95dd627ef356142 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Refusing update from remote peer 1526c5008f714bcfa69e9bacb51a662c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.632690 715 raft_consensus.cc:1261] T c99a78b4f41244a9a95dd627ef356142 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Refusing update from remote peer 1526c5008f714bcfa69e9bacb51a662c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.633407 1605 consensus_queue.cc:1035] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.633829 582 raft_consensus.cc:1261] T 27db9a40cf144c4a9018e906ee72e54f P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Refusing update from remote peer f5af98ef61b54ae0b33faeb78af96dec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.634363 847 raft_consensus.cc:1261] T 27db9a40cf144c4a9018e906ee72e54f P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Refusing update from remote peer f5af98ef61b54ae0b33faeb78af96dec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.634732 1746 consensus_queue.cc:1035] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.635037 1746 consensus_queue.cc:1035] T 27db9a40cf144c4a9018e906ee72e54f P f5af98ef61b54ae0b33faeb78af96dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.635888 1775 consensus_queue.cc:1035] T c99a78b4f41244a9a95dd627ef356142 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.638403 1243 raft_consensus.cc:1261] T 142a1237964a40d1b7e5ce852633be44 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Refusing update from remote peer 1526c5008f714bcfa69e9bacb51a662c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.639117 715 raft_consensus.cc:1261] T 142a1237964a40d1b7e5ce852633be44 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Refusing update from remote peer 1526c5008f714bcfa69e9bacb51a662c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.640937 1605 consensus_queue.cc:1035] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.642334 1605 consensus_queue.cc:1035] T 142a1237964a40d1b7e5ce852633be44 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 117.8 ms
time per row: 0.0589 ms
I20240501 07:59:32.710116 440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36176:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:32.715785 827 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0cf885cadd4d51960e4d198ed41679 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9101ca49cde44328957e165cd715168f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.716192 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0cf885cadd4d51960e4d198ed41679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.716336 563 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0cf885cadd4d51960e4d198ed41679 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9101ca49cde44328957e165cd715168f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.716620 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0cf885cadd4d51960e4d198ed41679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.716624 1091 tablet_service.cc:1467] Processing CreateTablet for tablet fbbab77fbfef40a08384f5d24526c8b4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9101ca49cde44328957e165cd715168f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.716910 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbbab77fbfef40a08384f5d24526c8b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.718851 1844 tablet_bootstrap.cc:492] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:32.718873 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0cf885cadd4d51960e4d198ed41679 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9101ca49cde44328957e165cd715168f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.719135 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0cf885cadd4d51960e4d198ed41679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.719080 695 tablet_service.cc:1467] Processing CreateTablet for tablet 7318525e5563450aa4875dc993637844 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9101ca49cde44328957e165cd715168f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.719339 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7318525e5563450aa4875dc993637844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.719641 1844 tablet_bootstrap.cc:654] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.720566 1844 tablet_bootstrap.cc:492] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:32.720628 1844 ts_tablet_manager.cc:1367] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:32.720960 1844 raft_consensus.cc:348] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.721050 1844 raft_consensus.cc:374] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.721081 1844 raft_consensus.cc:729] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.721287 1844 consensus_queue.cc:260] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.721578 1846 tablet_bootstrap.cc:492] T 1f0cf885cadd4d51960e4d198ed41679 P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:32.721782 1844 ts_tablet_manager.cc:1398] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:32.721853 826 tablet_service.cc:1467] Processing CreateTablet for tablet fbbab77fbfef40a08384f5d24526c8b4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9101ca49cde44328957e165cd715168f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.722095 1848 tablet_bootstrap.cc:492] T 7318525e5563450aa4875dc993637844 P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:32.722144 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbbab77fbfef40a08384f5d24526c8b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.722797 1846 tablet_bootstrap.cc:654] T 1f0cf885cadd4d51960e4d198ed41679 P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.723102 1848 tablet_bootstrap.cc:654] T 7318525e5563450aa4875dc993637844 P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.724429 1844 tablet_bootstrap.cc:492] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:32.724515 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 7318525e5563450aa4875dc993637844 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9101ca49cde44328957e165cd715168f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.724795 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7318525e5563450aa4875dc993637844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.725390 1844 tablet_bootstrap.cc:654] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.725545 959 tablet_service.cc:1467] Processing CreateTablet for tablet 7318525e5563450aa4875dc993637844 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9101ca49cde44328957e165cd715168f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.725860 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7318525e5563450aa4875dc993637844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.725958 1848 tablet_bootstrap.cc:492] T 7318525e5563450aa4875dc993637844 P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:32.726085 1848 ts_tablet_manager.cc:1367] T 7318525e5563450aa4875dc993637844 P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 07:59:32.726611 1844 tablet_bootstrap.cc:492] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:32.726680 1844 ts_tablet_manager.cc:1367] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:32.726601 1848 raft_consensus.cc:348] T 7318525e5563450aa4875dc993637844 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:32.726719 1848 raft_consensus.cc:374] T 7318525e5563450aa4875dc993637844 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.726750 1848 raft_consensus.cc:729] T 7318525e5563450aa4875dc993637844 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.726933 1848 consensus_queue.cc:260] T 7318525e5563450aa4875dc993637844 P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:32.727118 1844 raft_consensus.cc:348] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.727212 1844 raft_consensus.cc:374] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.727243 1844 raft_consensus.cc:729] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.727306 1848 ts_tablet_manager.cc:1398] T 7318525e5563450aa4875dc993637844 P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:32.727358 1844 consensus_queue.cc:260] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.727610 1844 ts_tablet_manager.cc:1398] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:32.728001 1852 tablet_bootstrap.cc:492] T 7318525e5563450aa4875dc993637844 P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:32.728967 1852 tablet_bootstrap.cc:654] T 7318525e5563450aa4875dc993637844 P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.728991 563 tablet_service.cc:1467] Processing CreateTablet for tablet fbbab77fbfef40a08384f5d24526c8b4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9101ca49cde44328957e165cd715168f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.729426 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbbab77fbfef40a08384f5d24526c8b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.729523 1853 tablet_bootstrap.cc:492] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:32.730655 1845 tablet_bootstrap.cc:492] T fbbab77fbfef40a08384f5d24526c8b4 P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:32.731202 1853 tablet_bootstrap.cc:654] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.731613 1845 tablet_bootstrap.cc:654] T fbbab77fbfef40a08384f5d24526c8b4 P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.732630 1853 tablet_bootstrap.cc:492] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:32.732731 1853 ts_tablet_manager.cc:1367] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:32.732741 1846 tablet_bootstrap.cc:492] T 1f0cf885cadd4d51960e4d198ed41679 P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:32.732888 1846 ts_tablet_manager.cc:1367] T 1f0cf885cadd4d51960e4d198ed41679 P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 07:59:32.732949 1845 tablet_bootstrap.cc:492] T fbbab77fbfef40a08384f5d24526c8b4 P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:32.733040 1845 ts_tablet_manager.cc:1367] T fbbab77fbfef40a08384f5d24526c8b4 P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:32.733217 1853 raft_consensus.cc:348] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:32.733366 1853 raft_consensus.cc:374] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.733310 1846 raft_consensus.cc:348] T 1f0cf885cadd4d51960e4d198ed41679 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.733402 1853 raft_consensus.cc:729] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.733413 1846 raft_consensus.cc:374] T 1f0cf885cadd4d51960e4d198ed41679 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.733443 1846 raft_consensus.cc:729] T 1f0cf885cadd4d51960e4d198ed41679 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.733580 1846 consensus_queue.cc:260] T 1f0cf885cadd4d51960e4d198ed41679 P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.733628 1853 consensus_queue.cc:260] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:32.733891 1846 ts_tablet_manager.cc:1398] T 1f0cf885cadd4d51960e4d198ed41679 P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:32.733930 1845 raft_consensus.cc:348] T fbbab77fbfef40a08384f5d24526c8b4 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.734066 1845 raft_consensus.cc:374] T fbbab77fbfef40a08384f5d24526c8b4 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.734107 1845 raft_consensus.cc:729] T fbbab77fbfef40a08384f5d24526c8b4 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.734207 1846 tablet_bootstrap.cc:492] T fbbab77fbfef40a08384f5d24526c8b4 P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:32.734251 1845 consensus_queue.cc:260] T fbbab77fbfef40a08384f5d24526c8b4 P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.734513 1845 ts_tablet_manager.cc:1398] T fbbab77fbfef40a08384f5d24526c8b4 P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:32.734660 1845 tablet_bootstrap.cc:492] T 1f0cf885cadd4d51960e4d198ed41679 P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:32.735204 1846 tablet_bootstrap.cc:654] T fbbab77fbfef40a08384f5d24526c8b4 P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.735538 1845 tablet_bootstrap.cc:654] T 1f0cf885cadd4d51960e4d198ed41679 P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.736949 1845 tablet_bootstrap.cc:492] T 1f0cf885cadd4d51960e4d198ed41679 P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:32.737049 1845 ts_tablet_manager.cc:1367] T 1f0cf885cadd4d51960e4d198ed41679 P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:32.737345 1853 ts_tablet_manager.cc:1398] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:59:32.737555 1845 raft_consensus.cc:348] T 1f0cf885cadd4d51960e4d198ed41679 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.737676 1845 raft_consensus.cc:374] T 1f0cf885cadd4d51960e4d198ed41679 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.737712 1845 raft_consensus.cc:729] T 1f0cf885cadd4d51960e4d198ed41679 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.738157 1852 tablet_bootstrap.cc:492] T 7318525e5563450aa4875dc993637844 P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:32.738262 1852 ts_tablet_manager.cc:1367] T 7318525e5563450aa4875dc993637844 P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240501 07:59:32.738382 1846 tablet_bootstrap.cc:492] T fbbab77fbfef40a08384f5d24526c8b4 P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:32.738325 1845 consensus_queue.cc:260] T 1f0cf885cadd4d51960e4d198ed41679 P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.738452 1846 ts_tablet_manager.cc:1367] T fbbab77fbfef40a08384f5d24526c8b4 P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:32.738775 1845 ts_tablet_manager.cc:1398] T 1f0cf885cadd4d51960e4d198ed41679 P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:32.738759 1852 raft_consensus.cc:348] T 7318525e5563450aa4875dc993637844 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:32.738873 1852 raft_consensus.cc:374] T 7318525e5563450aa4875dc993637844 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.738902 1852 raft_consensus.cc:729] T 7318525e5563450aa4875dc993637844 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.739018 1852 consensus_queue.cc:260] T 7318525e5563450aa4875dc993637844 P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:32.739300 1852 ts_tablet_manager.cc:1398] T 7318525e5563450aa4875dc993637844 P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:32.740731 1846 raft_consensus.cc:348] T fbbab77fbfef40a08384f5d24526c8b4 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.740893 1846 raft_consensus.cc:374] T fbbab77fbfef40a08384f5d24526c8b4 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.740933 1846 raft_consensus.cc:729] T fbbab77fbfef40a08384f5d24526c8b4 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.741045 1846 consensus_queue.cc:260] T fbbab77fbfef40a08384f5d24526c8b4 P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.741303 1846 ts_tablet_manager.cc:1398] T fbbab77fbfef40a08384f5d24526c8b4 P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:32.854007 1740 raft_consensus.cc:482] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:32.854161 1740 raft_consensus.cc:504] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.854624 1740 leader_election.cc:288] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:32.855109 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbbab77fbfef40a08384f5d24526c8b4" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec" is_pre_election: true
I20240501 07:59:32.855114 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbbab77fbfef40a08384f5d24526c8b4" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747" is_pre_election: true
I20240501 07:59:32.855260 1106 raft_consensus.cc:2454] T fbbab77fbfef40a08384f5d24526c8b4 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:32.855608 780 leader_election.cc:302] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:32.855276 582 raft_consensus.cc:2454] T fbbab77fbfef40a08384f5d24526c8b4 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:32.855784 1740 raft_consensus.cc:2781] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:32.855835 1740 raft_consensus.cc:482] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:32.855891 1740 raft_consensus.cc:3037] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.856750 1740 raft_consensus.cc:504] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.857105 1740 leader_election.cc:288] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Requested vote from peers b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:32.857481 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbbab77fbfef40a08384f5d24526c8b4" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747"
I20240501 07:59:32.857481 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbbab77fbfef40a08384f5d24526c8b4" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec"
I20240501 07:59:32.857609 1106 raft_consensus.cc:3037] T fbbab77fbfef40a08384f5d24526c8b4 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.857609 582 raft_consensus.cc:3037] T fbbab77fbfef40a08384f5d24526c8b4 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.858631 582 raft_consensus.cc:2454] T fbbab77fbfef40a08384f5d24526c8b4 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:32.858631 1106 raft_consensus.cc:2454] T fbbab77fbfef40a08384f5d24526c8b4 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:32.858963 781 leader_election.cc:302] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, b9644aeae9de4deba0c3a2b85df04747; no voters:
I20240501 07:59:32.859112 1740 raft_consensus.cc:2781] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:32.859326 1740 raft_consensus.cc:686] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee66e185465431389043d933cb51308, State: Running, Role: LEADER
I20240501 07:59:32.859494 1740 consensus_queue.cc:237] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.861413 432 catalog_manager.cc:5484] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee66e185465431389043d933cb51308 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8ee66e185465431389043d933cb51308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:32.949941 1740 raft_consensus.cc:482] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:32.950119 1740 raft_consensus.cc:504] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.950570 1740 leader_election.cc:288] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:32.951046 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0cf885cadd4d51960e4d198ed41679" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec" is_pre_election: true
I20240501 07:59:32.951050 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0cf885cadd4d51960e4d198ed41679" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747" is_pre_election: true
I20240501 07:59:32.951200 1106 raft_consensus.cc:2454] T 1f0cf885cadd4d51960e4d198ed41679 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:32.951200 582 raft_consensus.cc:2454] T 1f0cf885cadd4d51960e4d198ed41679 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:32.951507 781 leader_election.cc:302] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, b9644aeae9de4deba0c3a2b85df04747; no voters:
I20240501 07:59:32.951687 1740 raft_consensus.cc:2781] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:32.951753 1740 raft_consensus.cc:482] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:32.951826 1740 raft_consensus.cc:3037] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.952594 1740 raft_consensus.cc:504] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.952915 1740 leader_election.cc:288] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Requested vote from peers b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:32.953254 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0cf885cadd4d51960e4d198ed41679" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec"
I20240501 07:59:32.953254 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0cf885cadd4d51960e4d198ed41679" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747"
I20240501 07:59:32.953387 1106 raft_consensus.cc:3037] T 1f0cf885cadd4d51960e4d198ed41679 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.953406 582 raft_consensus.cc:3037] T 1f0cf885cadd4d51960e4d198ed41679 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.954384 1106 raft_consensus.cc:2454] T 1f0cf885cadd4d51960e4d198ed41679 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:32.954384 582 raft_consensus.cc:2454] T 1f0cf885cadd4d51960e4d198ed41679 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:32.954695 780 leader_election.cc:302] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:32.954892 1740 raft_consensus.cc:2781] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:32.954975 1740 raft_consensus.cc:686] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee66e185465431389043d933cb51308, State: Running, Role: LEADER
I20240501 07:59:32.955118 1740 consensus_queue.cc:237] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:32.957060 432 catalog_manager.cc:5484] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee66e185465431389043d933cb51308 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8ee66e185465431389043d933cb51308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:33.042591 1816 raft_consensus.cc:482] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:33.042730 1816 raft_consensus.cc:504] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:33.043175 1816 leader_election.cc:288] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:33.043587 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7318525e5563450aa4875dc993637844" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725" is_pre_election: true
I20240501 07:59:33.043627 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7318525e5563450aa4875dc993637844" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429" is_pre_election: true
I20240501 07:59:33.043738 715 raft_consensus.cc:2454] T 7318525e5563450aa4875dc993637844 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:33.043776 1242 raft_consensus.cc:2454] T 7318525e5563450aa4875dc993637844 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:33.044095 912 leader_election.cc:302] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:33.044267 1816 raft_consensus.cc:2781] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:33.044332 1816 raft_consensus.cc:482] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:33.044359 1816 raft_consensus.cc:3037] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.045161 1816 raft_consensus.cc:504] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:33.045475 1816 leader_election.cc:288] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Requested vote from peers 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:33.045876 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7318525e5563450aa4875dc993637844" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725"
I20240501 07:59:33.045897 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7318525e5563450aa4875dc993637844" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429"
I20240501 07:59:33.046022 1242 raft_consensus.cc:3037] T 7318525e5563450aa4875dc993637844 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.046020 715 raft_consensus.cc:3037] T 7318525e5563450aa4875dc993637844 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.046759 1242 raft_consensus.cc:2454] T 7318525e5563450aa4875dc993637844 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:33.046762 715 raft_consensus.cc:2454] T 7318525e5563450aa4875dc993637844 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:33.047026 912 leader_election.cc:302] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:33.047235 1816 raft_consensus.cc:2781] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:33.047317 1816 raft_consensus.cc:686] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [term 1 LEADER]: Becoming Leader. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Running, Role: LEADER
I20240501 07:59:33.047462 1816 consensus_queue.cc:237] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:33.049417 432 catalog_manager.cc:5484] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1526c5008f714bcfa69e9bacb51a662c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: HEALTHY } } }
W20240501 07:59:33.203284 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:33.203644 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:33.207194 1863 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 07:59:33.324285 1106 raft_consensus.cc:1261] T fbbab77fbfef40a08384f5d24526c8b4 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Refusing update from remote peer 8ee66e185465431389043d933cb51308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:33.324278 582 raft_consensus.cc:1261] T fbbab77fbfef40a08384f5d24526c8b4 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Refusing update from remote peer 8ee66e185465431389043d933cb51308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:33.324817 1740 consensus_queue.cc:1035] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:33.325075 1859 consensus_queue.cc:1035] T fbbab77fbfef40a08384f5d24526c8b4 P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:33.326812 714 raft_consensus.cc:1261] T 7318525e5563450aa4875dc993637844 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Refusing update from remote peer 1526c5008f714bcfa69e9bacb51a662c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:33.327394 1816 consensus_queue.cc:1035] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:33.327876 1242 raft_consensus.cc:1261] T 7318525e5563450aa4875dc993637844 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Refusing update from remote peer 1526c5008f714bcfa69e9bacb51a662c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:33.329347 1816 consensus_queue.cc:1035] T 7318525e5563450aa4875dc993637844 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:33.341557 583 raft_consensus.cc:1261] T 1f0cf885cadd4d51960e4d198ed41679 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Refusing update from remote peer 8ee66e185465431389043d933cb51308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:33.342208 1740 consensus_queue.cc:1035] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:33.342659 1106 raft_consensus.cc:1261] T 1f0cf885cadd4d51960e4d198ed41679 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Refusing update from remote peer 8ee66e185465431389043d933cb51308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:33.344240 1859 consensus_queue.cc:1035] T 1f0cf885cadd4d51960e4d198ed41679 P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 141.385 ms
time per row: 0.0706925 ms
I20240501 07:59:33.399820 438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36176:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:33.405825 959 tablet_service.cc:1467] Processing CreateTablet for tablet 9d2047f2e0094e99842cc3929571f39c (DEFAULT_TABLE table=rebalance_test_table_7 [id=3ec4f48452134dbaa98b6a7fa97ffff4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.406204 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d2047f2e0094e99842cc3929571f39c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.406340 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 9d2047f2e0094e99842cc3929571f39c (DEFAULT_TABLE table=rebalance_test_table_7 [id=3ec4f48452134dbaa98b6a7fa97ffff4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.406621 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d2047f2e0094e99842cc3929571f39c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.408732 1922 tablet_bootstrap.cc:492] T 9d2047f2e0094e99842cc3929571f39c P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:33.408926 695 tablet_service.cc:1467] Processing CreateTablet for tablet 9d2047f2e0094e99842cc3929571f39c (DEFAULT_TABLE table=rebalance_test_table_7 [id=3ec4f48452134dbaa98b6a7fa97ffff4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.409235 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d2047f2e0094e99842cc3929571f39c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.409698 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 45765892e3df4768b034a748ec5e0980 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3ec4f48452134dbaa98b6a7fa97ffff4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.409940 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45765892e3df4768b034a748ec5e0980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.410812 694 tablet_service.cc:1467] Processing CreateTablet for tablet ead49854fd4d484db8b90d1e14ec3873 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3ec4f48452134dbaa98b6a7fa97ffff4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.411003 1923 tablet_bootstrap.cc:492] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:33.411156 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead49854fd4d484db8b90d1e14ec3873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.411828 563 tablet_service.cc:1467] Processing CreateTablet for tablet 45765892e3df4768b034a748ec5e0980 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3ec4f48452134dbaa98b6a7fa97ffff4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.412205 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45765892e3df4768b034a748ec5e0980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.412402 1923 tablet_bootstrap.cc:654] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.408959 1223 tablet_service.cc:1467] Processing CreateTablet for tablet ead49854fd4d484db8b90d1e14ec3873 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3ec4f48452134dbaa98b6a7fa97ffff4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.413419 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead49854fd4d484db8b90d1e14ec3873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.413494 1922 tablet_bootstrap.cc:654] T 9d2047f2e0094e99842cc3929571f39c P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.414417 826 tablet_service.cc:1467] Processing CreateTablet for tablet 45765892e3df4768b034a748ec5e0980 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3ec4f48452134dbaa98b6a7fa97ffff4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.414566 1923 tablet_bootstrap.cc:492] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:33.414566 1922 tablet_bootstrap.cc:492] T 9d2047f2e0094e99842cc3929571f39c P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:33.414646 1923 ts_tablet_manager.cc:1367] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:33.414665 1922 ts_tablet_manager.cc:1367] T 9d2047f2e0094e99842cc3929571f39c P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:33.414678 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45765892e3df4768b034a748ec5e0980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.415093 1922 raft_consensus.cc:348] T 9d2047f2e0094e99842cc3929571f39c P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:33.415093 1923 raft_consensus.cc:348] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:33.415205 1923 raft_consensus.cc:374] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.415205 1922 raft_consensus.cc:374] T 9d2047f2e0094e99842cc3929571f39c P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.415237 1923 raft_consensus.cc:729] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.415237 1922 raft_consensus.cc:729] T 9d2047f2e0094e99842cc3929571f39c P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.415388 1922 consensus_queue.cc:260] T 9d2047f2e0094e99842cc3929571f39c P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:33.415422 1923 consensus_queue.cc:260] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:33.415668 1922 ts_tablet_manager.cc:1398] T 9d2047f2e0094e99842cc3929571f39c P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:33.416327 1923 ts_tablet_manager.cc:1398] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:33.416496 1923 tablet_bootstrap.cc:492] T ead49854fd4d484db8b90d1e14ec3873 P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:33.417538 1923 tablet_bootstrap.cc:654] T ead49854fd4d484db8b90d1e14ec3873 P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.418190 959 tablet_service.cc:1467] Processing CreateTablet for tablet ead49854fd4d484db8b90d1e14ec3873 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3ec4f48452134dbaa98b6a7fa97ffff4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.418448 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead49854fd4d484db8b90d1e14ec3873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.418581 1925 tablet_bootstrap.cc:492] T 45765892e3df4768b034a748ec5e0980 P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:33.418819 1923 tablet_bootstrap.cc:492] T ead49854fd4d484db8b90d1e14ec3873 P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:33.418924 1923 ts_tablet_manager.cc:1367] T ead49854fd4d484db8b90d1e14ec3873 P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:33.419373 1923 raft_consensus.cc:348] T ead49854fd4d484db8b90d1e14ec3873 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:33.419503 1923 raft_consensus.cc:374] T ead49854fd4d484db8b90d1e14ec3873 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.419565 1923 raft_consensus.cc:729] T ead49854fd4d484db8b90d1e14ec3873 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.419699 1923 consensus_queue.cc:260] T ead49854fd4d484db8b90d1e14ec3873 P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:33.419939 1925 tablet_bootstrap.cc:654] T 45765892e3df4768b034a748ec5e0980 P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.419994 1923 ts_tablet_manager.cc:1398] T ead49854fd4d484db8b90d1e14ec3873 P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:33.420540 1922 tablet_bootstrap.cc:492] T ead49854fd4d484db8b90d1e14ec3873 P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:33.420619 1928 tablet_bootstrap.cc:492] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:33.421569 1922 tablet_bootstrap.cc:654] T ead49854fd4d484db8b90d1e14ec3873 P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.421599 1928 tablet_bootstrap.cc:654] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.422561 1925 tablet_bootstrap.cc:492] T 45765892e3df4768b034a748ec5e0980 P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:33.422643 1929 tablet_bootstrap.cc:492] T 45765892e3df4768b034a748ec5e0980 P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:33.422663 1925 ts_tablet_manager.cc:1367] T 45765892e3df4768b034a748ec5e0980 P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:33.423085 1928 tablet_bootstrap.cc:492] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:33.423170 1928 ts_tablet_manager.cc:1367] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:33.423172 1925 raft_consensus.cc:348] T 45765892e3df4768b034a748ec5e0980 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:33.423303 1925 raft_consensus.cc:374] T 45765892e3df4768b034a748ec5e0980 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.423337 1925 raft_consensus.cc:729] T 45765892e3df4768b034a748ec5e0980 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.423460 1925 consensus_queue.cc:260] T 45765892e3df4768b034a748ec5e0980 P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:33.423607 1929 tablet_bootstrap.cc:654] T 45765892e3df4768b034a748ec5e0980 P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.423655 1928 raft_consensus.cc:348] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:33.423761 1925 ts_tablet_manager.cc:1398] T 45765892e3df4768b034a748ec5e0980 P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:33.423760 1928 raft_consensus.cc:374] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.423801 1928 raft_consensus.cc:729] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.423923 1928 consensus_queue.cc:260] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:33.424245 1928 ts_tablet_manager.cc:1398] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:33.424319 1924 tablet_bootstrap.cc:492] T 9d2047f2e0094e99842cc3929571f39c P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:33.425323 1924 tablet_bootstrap.cc:654] T 9d2047f2e0094e99842cc3929571f39c P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.426283 1922 tablet_bootstrap.cc:492] T ead49854fd4d484db8b90d1e14ec3873 P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:33.426379 1922 ts_tablet_manager.cc:1367] T ead49854fd4d484db8b90d1e14ec3873 P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 07:59:33.426626 1929 tablet_bootstrap.cc:492] T 45765892e3df4768b034a748ec5e0980 P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:33.426630 1924 tablet_bootstrap.cc:492] T 9d2047f2e0094e99842cc3929571f39c P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:33.426714 1929 ts_tablet_manager.cc:1367] T 45765892e3df4768b034a748ec5e0980 P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:33.426715 1924 ts_tablet_manager.cc:1367] T 9d2047f2e0094e99842cc3929571f39c P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:33.426856 1922 raft_consensus.cc:348] T ead49854fd4d484db8b90d1e14ec3873 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:33.426970 1922 raft_consensus.cc:374] T ead49854fd4d484db8b90d1e14ec3873 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.427002 1922 raft_consensus.cc:729] T ead49854fd4d484db8b90d1e14ec3873 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.427129 1922 consensus_queue.cc:260] T ead49854fd4d484db8b90d1e14ec3873 P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:33.427174 1924 raft_consensus.cc:348] T 9d2047f2e0094e99842cc3929571f39c P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:33.427264 1924 raft_consensus.cc:374] T 9d2047f2e0094e99842cc3929571f39c P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.427182 1929 raft_consensus.cc:348] T 45765892e3df4768b034a748ec5e0980 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:33.427294 1924 raft_consensus.cc:729] T 9d2047f2e0094e99842cc3929571f39c P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.427304 1929 raft_consensus.cc:374] T 45765892e3df4768b034a748ec5e0980 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.427341 1929 raft_consensus.cc:729] T 45765892e3df4768b034a748ec5e0980 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.427428 1922 ts_tablet_manager.cc:1398] T ead49854fd4d484db8b90d1e14ec3873 P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:33.427475 1929 consensus_queue.cc:260] T 45765892e3df4768b034a748ec5e0980 P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:33.427480 1924 consensus_queue.cc:260] T 9d2047f2e0094e99842cc3929571f39c P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:33.428043 1924 ts_tablet_manager.cc:1398] T 9d2047f2e0094e99842cc3929571f39c P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:33.428301 1929 ts_tablet_manager.cc:1398] T 45765892e3df4768b034a748ec5e0980 P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:33.428386 1924 tablet_bootstrap.cc:492] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:33.430444 1924 tablet_bootstrap.cc:654] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.431820 1924 tablet_bootstrap.cc:492] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:33.431898 1924 ts_tablet_manager.cc:1367] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:33.432278 1924 raft_consensus.cc:348] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:33.432387 1924 raft_consensus.cc:374] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.432410 1924 raft_consensus.cc:729] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.432540 1924 consensus_queue.cc:260] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:33.432808 1924 ts_tablet_manager.cc:1398] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:33.494220 1740 raft_consensus.cc:482] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:33.494359 1740 raft_consensus.cc:504] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:33.494777 1740 leader_election.cc:288] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665), b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609)
I20240501 07:59:33.495201 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45765892e3df4768b034a748ec5e0980" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec" is_pre_election: true
I20240501 07:59:33.495203 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45765892e3df4768b034a748ec5e0980" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747" is_pre_election: true
I20240501 07:59:33.495360 1106 raft_consensus.cc:2454] T 45765892e3df4768b034a748ec5e0980 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:33.495359 582 raft_consensus.cc:2454] T 45765892e3df4768b034a748ec5e0980 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:33.495638 781 leader_election.cc:302] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, b9644aeae9de4deba0c3a2b85df04747; no voters:
I20240501 07:59:33.495810 1740 raft_consensus.cc:2781] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:33.495877 1740 raft_consensus.cc:482] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:33.495909 1740 raft_consensus.cc:3037] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.496785 1740 raft_consensus.cc:504] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:33.497182 1740 leader_election.cc:288] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Requested vote from peers f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665), b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609)
I20240501 07:59:33.497489 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45765892e3df4768b034a748ec5e0980" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec"
I20240501 07:59:33.497614 1106 raft_consensus.cc:3037] T 45765892e3df4768b034a748ec5e0980 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.497671 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45765892e3df4768b034a748ec5e0980" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747"
I20240501 07:59:33.497968 582 raft_consensus.cc:3037] T 45765892e3df4768b034a748ec5e0980 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.498589 1106 raft_consensus.cc:2454] T 45765892e3df4768b034a748ec5e0980 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:33.498840 582 raft_consensus.cc:2454] T 45765892e3df4768b034a748ec5e0980 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:33.498955 780 leader_election.cc:302] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:33.499259 1740 raft_consensus.cc:2781] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:33.499364 1740 raft_consensus.cc:686] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee66e185465431389043d933cb51308, State: Running, Role: LEADER
I20240501 07:59:33.499579 1740 consensus_queue.cc:237] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:33.501555 441 catalog_manager.cc:5484] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee66e185465431389043d933cb51308 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8ee66e185465431389043d933cb51308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:33.556305 1520 raft_consensus.cc:482] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:33.556445 1520 raft_consensus.cc:504] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:33.556834 1520 leader_election.cc:288] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663), 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765)
I20240501 07:59:33.557533 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2047f2e0094e99842cc3929571f39c" candidate_uuid: "c45c306768564fb2bd741c535d334429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1526c5008f714bcfa69e9bacb51a662c" is_pre_election: true
I20240501 07:59:33.557510 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2047f2e0094e99842cc3929571f39c" candidate_uuid: "c45c306768564fb2bd741c535d334429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725" is_pre_election: true
I20240501 07:59:33.557694 714 raft_consensus.cc:2454] T 9d2047f2e0094e99842cc3929571f39c P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c45c306768564fb2bd741c535d334429 in term 0.
I20240501 07:59:33.557694 978 raft_consensus.cc:2454] T 9d2047f2e0094e99842cc3929571f39c P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c45c306768564fb2bd741c535d334429 in term 0.
I20240501 07:59:33.558073 1177 leader_election.cc:302] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee011df676c43f7beaeace0f8c60725, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:33.558260 1520 raft_consensus.cc:2781] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:33.558331 1520 raft_consensus.cc:482] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:33.558359 1520 raft_consensus.cc:3037] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.559239 1520 raft_consensus.cc:504] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:33.559592 1520 leader_election.cc:288] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [CANDIDATE]: Term 1 election: Requested vote from peers 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663), 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765)
I20240501 07:59:33.560026 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2047f2e0094e99842cc3929571f39c" candidate_uuid: "c45c306768564fb2bd741c535d334429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1526c5008f714bcfa69e9bacb51a662c"
I20240501 07:59:33.560081 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2047f2e0094e99842cc3929571f39c" candidate_uuid: "c45c306768564fb2bd741c535d334429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725"
I20240501 07:59:33.560190 714 raft_consensus.cc:3037] T 9d2047f2e0094e99842cc3929571f39c P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.560191 978 raft_consensus.cc:3037] T 9d2047f2e0094e99842cc3929571f39c P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.561270 714 raft_consensus.cc:2454] T 9d2047f2e0094e99842cc3929571f39c P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c45c306768564fb2bd741c535d334429 in term 1.
I20240501 07:59:33.561270 978 raft_consensus.cc:2454] T 9d2047f2e0094e99842cc3929571f39c P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c45c306768564fb2bd741c535d334429 in term 1.
I20240501 07:59:33.561604 1177 leader_election.cc:302] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:33.561767 1520 raft_consensus.cc:2781] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:33.562009 1520 raft_consensus.cc:686] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [term 1 LEADER]: Becoming Leader. State: Replica: c45c306768564fb2bd741c535d334429, State: Running, Role: LEADER
I20240501 07:59:33.562180 1520 consensus_queue.cc:237] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:33.564157 441 catalog_manager.cc:5484] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 reported cstate change: term changed from 0 to 1, leader changed from <none> to c45c306768564fb2bd741c535d334429 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "c45c306768564fb2bd741c535d334429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:33.707180 1645 raft_consensus.cc:482] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:33.707337 1645 raft_consensus.cc:504] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:33.707779 1645 leader_election.cc:288] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:33.708246 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead49854fd4d484db8b90d1e14ec3873" candidate_uuid: "3ee011df676c43f7beaeace0f8c60725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1526c5008f714bcfa69e9bacb51a662c" is_pre_election: true
I20240501 07:59:33.708262 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead49854fd4d484db8b90d1e14ec3873" candidate_uuid: "3ee011df676c43f7beaeace0f8c60725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429" is_pre_election: true
I20240501 07:59:33.708391 978 raft_consensus.cc:2454] T ead49854fd4d484db8b90d1e14ec3873 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ee011df676c43f7beaeace0f8c60725 in term 0.
I20240501 07:59:33.708421 1242 raft_consensus.cc:2454] T ead49854fd4d484db8b90d1e14ec3873 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ee011df676c43f7beaeace0f8c60725 in term 0.
I20240501 07:59:33.708740 649 leader_election.cc:302] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, 3ee011df676c43f7beaeace0f8c60725; no voters:
I20240501 07:59:33.708923 1645 raft_consensus.cc:2781] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:33.708981 1645 raft_consensus.cc:482] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:33.709034 1645 raft_consensus.cc:3037] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.710206 1645 raft_consensus.cc:504] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:33.710541 1645 leader_election.cc:288] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [CANDIDATE]: Term 1 election: Requested vote from peers 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:33.710933 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead49854fd4d484db8b90d1e14ec3873" candidate_uuid: "3ee011df676c43f7beaeace0f8c60725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429"
I20240501 07:59:33.710948 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead49854fd4d484db8b90d1e14ec3873" candidate_uuid: "3ee011df676c43f7beaeace0f8c60725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1526c5008f714bcfa69e9bacb51a662c"
I20240501 07:59:33.711066 1242 raft_consensus.cc:3037] T ead49854fd4d484db8b90d1e14ec3873 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.711086 978 raft_consensus.cc:3037] T ead49854fd4d484db8b90d1e14ec3873 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.712030 1242 raft_consensus.cc:2454] T ead49854fd4d484db8b90d1e14ec3873 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ee011df676c43f7beaeace0f8c60725 in term 1.
I20240501 07:59:33.712029 978 raft_consensus.cc:2454] T ead49854fd4d484db8b90d1e14ec3873 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ee011df676c43f7beaeace0f8c60725 in term 1.
I20240501 07:59:33.712357 648 leader_election.cc:302] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee011df676c43f7beaeace0f8c60725, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:33.712515 1645 raft_consensus.cc:2781] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:33.712671 1645 raft_consensus.cc:686] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [term 1 LEADER]: Becoming Leader. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Running, Role: LEADER
I20240501 07:59:33.712826 1645 consensus_queue.cc:237] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:33.714968 441 catalog_manager.cc:5484] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3ee011df676c43f7beaeace0f8c60725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: HEALTHY } } }
W20240501 07:59:33.904582 1942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:33.904891 1942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:33.908239 1942 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 07:59:34.008252 715 raft_consensus.cc:1261] T 9d2047f2e0094e99842cc3929571f39c P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Refusing update from remote peer c45c306768564fb2bd741c535d334429: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.008890 1520 consensus_queue.cc:1035] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.010738 582 raft_consensus.cc:1261] T 45765892e3df4768b034a748ec5e0980 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Refusing update from remote peer 8ee66e185465431389043d933cb51308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.011148 1106 raft_consensus.cc:1261] T 45765892e3df4768b034a748ec5e0980 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Refusing update from remote peer 8ee66e185465431389043d933cb51308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.011201 1977 consensus_queue.cc:1035] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.011648 1740 consensus_queue.cc:1035] T 45765892e3df4768b034a748ec5e0980 P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.016722 978 raft_consensus.cc:1261] T 9d2047f2e0094e99842cc3929571f39c P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Refusing update from remote peer c45c306768564fb2bd741c535d334429: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.017634 1520 consensus_queue.cc:1035] T 9d2047f2e0094e99842cc3929571f39c P c45c306768564fb2bd741c535d334429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.025051 1242 raft_consensus.cc:1261] T ead49854fd4d484db8b90d1e14ec3873 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Refusing update from remote peer 3ee011df676c43f7beaeace0f8c60725: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.025307 979 raft_consensus.cc:1261] T ead49854fd4d484db8b90d1e14ec3873 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Refusing update from remote peer 3ee011df676c43f7beaeace0f8c60725: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.025835 1940 consensus_queue.cc:1035] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.026209 1940 consensus_queue.cc:1035] T ead49854fd4d484db8b90d1e14ec3873 P 3ee011df676c43f7beaeace0f8c60725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 136.414 ms
time per row: 0.0682069 ms
I20240501 07:59:34.091545 435 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36176:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:34.097527 563 tablet_service.cc:1467] Processing CreateTablet for tablet 37732050f7a74053827b657e521f9a92 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8af9fc6b11c34108834bb0dc831385f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.097609 826 tablet_service.cc:1467] Processing CreateTablet for tablet 37732050f7a74053827b657e521f9a92 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8af9fc6b11c34108834bb0dc831385f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.097877 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37732050f7a74053827b657e521f9a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.097892 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37732050f7a74053827b657e521f9a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.098320 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 37732050f7a74053827b657e521f9a92 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8af9fc6b11c34108834bb0dc831385f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.098577 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37732050f7a74053827b657e521f9a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.100279 2007 tablet_bootstrap.cc:492] T 37732050f7a74053827b657e521f9a92 P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:34.100601 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5c68ac4b29424e8f0dbb50dbdfb77f (DEFAULT_TABLE table=rebalance_test_table_8 [id=8af9fc6b11c34108834bb0dc831385f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.100647 1223 tablet_service.cc:1467] Processing CreateTablet for tablet ac6730a31bab4499a2c97f95ea217a09 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8af9fc6b11c34108834bb0dc831385f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.100858 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5c68ac4b29424e8f0dbb50dbdfb77f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.100908 2009 tablet_bootstrap.cc:492] T 37732050f7a74053827b657e521f9a92 P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:34.100939 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac6730a31bab4499a2c97f95ea217a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.101475 2008 tablet_bootstrap.cc:492] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:34.101789 2009 tablet_bootstrap.cc:654] T 37732050f7a74053827b657e521f9a92 P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.102466 2008 tablet_bootstrap.cc:654] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.102764 563 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5c68ac4b29424e8f0dbb50dbdfb77f (DEFAULT_TABLE table=rebalance_test_table_8 [id=8af9fc6b11c34108834bb0dc831385f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.103010 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5c68ac4b29424e8f0dbb50dbdfb77f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.103152 694 tablet_service.cc:1467] Processing CreateTablet for tablet ac6730a31bab4499a2c97f95ea217a09 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8af9fc6b11c34108834bb0dc831385f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.103181 959 tablet_service.cc:1467] Processing CreateTablet for tablet ac6730a31bab4499a2c97f95ea217a09 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8af9fc6b11c34108834bb0dc831385f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.103442 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac6730a31bab4499a2c97f95ea217a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.103464 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac6730a31bab4499a2c97f95ea217a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.103837 2007 tablet_bootstrap.cc:654] T 37732050f7a74053827b657e521f9a92 P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.105119 2008 tablet_bootstrap.cc:492] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:34.105199 2008 ts_tablet_manager.cc:1367] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:34.105657 2013 tablet_bootstrap.cc:492] T ac6730a31bab4499a2c97f95ea217a09 P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:34.105680 2008 raft_consensus.cc:348] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.105803 2008 raft_consensus.cc:374] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.105831 2008 raft_consensus.cc:729] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.105965 2008 consensus_queue.cc:260] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.106269 2008 ts_tablet_manager.cc:1398] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:34.106442 2008 tablet_bootstrap.cc:492] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:34.106670 2013 tablet_bootstrap.cc:654] T ac6730a31bab4499a2c97f95ea217a09 P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.107244 2008 tablet_bootstrap.cc:654] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.107849 2013 tablet_bootstrap.cc:492] T ac6730a31bab4499a2c97f95ea217a09 P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:34.107937 2013 ts_tablet_manager.cc:1367] T ac6730a31bab4499a2c97f95ea217a09 P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:34.107923 2016 tablet_bootstrap.cc:492] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:34.108345 826 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5c68ac4b29424e8f0dbb50dbdfb77f (DEFAULT_TABLE table=rebalance_test_table_8 [id=8af9fc6b11c34108834bb0dc831385f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.108361 2013 raft_consensus.cc:348] T ac6730a31bab4499a2c97f95ea217a09 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.108453 2013 raft_consensus.cc:374] T ac6730a31bab4499a2c97f95ea217a09 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.108484 2013 raft_consensus.cc:729] T ac6730a31bab4499a2c97f95ea217a09 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.108616 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5c68ac4b29424e8f0dbb50dbdfb77f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.108603 2013 consensus_queue.cc:260] T ac6730a31bab4499a2c97f95ea217a09 P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.108696 2007 tablet_bootstrap.cc:492] T 37732050f7a74053827b657e521f9a92 P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:34.108778 2007 ts_tablet_manager.cc:1367] T 37732050f7a74053827b657e521f9a92 P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:34.108870 2013 ts_tablet_manager.cc:1398] T ac6730a31bab4499a2c97f95ea217a09 P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:34.109145 2016 tablet_bootstrap.cc:654] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.109238 2007 raft_consensus.cc:348] T 37732050f7a74053827b657e521f9a92 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.109345 2007 raft_consensus.cc:374] T 37732050f7a74053827b657e521f9a92 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.109375 2007 raft_consensus.cc:729] T 37732050f7a74053827b657e521f9a92 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.109515 2007 consensus_queue.cc:260] T 37732050f7a74053827b657e521f9a92 P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.109793 2007 ts_tablet_manager.cc:1398] T 37732050f7a74053827b657e521f9a92 P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:34.110296 2016 tablet_bootstrap.cc:492] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:34.110381 2016 ts_tablet_manager.cc:1367] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:34.110795 2016 raft_consensus.cc:348] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.110901 2016 raft_consensus.cc:374] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.110941 2016 raft_consensus.cc:729] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.111060 2016 consensus_queue.cc:260] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.111340 2016 ts_tablet_manager.cc:1398] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:34.113847 2007 tablet_bootstrap.cc:492] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:34.114568 2008 tablet_bootstrap.cc:492] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:34.114638 2008 ts_tablet_manager.cc:1367] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240501 07:59:34.114784 2007 tablet_bootstrap.cc:654] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.115012 2008 raft_consensus.cc:348] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:34.115109 2008 raft_consensus.cc:374] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.115139 2008 raft_consensus.cc:729] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.115252 2008 consensus_queue.cc:260] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:34.115562 2008 ts_tablet_manager.cc:1398] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:59:34.115948 2014 tablet_bootstrap.cc:492] T ac6730a31bab4499a2c97f95ea217a09 P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:34.116077 2007 tablet_bootstrap.cc:492] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:34.116240 2007 ts_tablet_manager.cc:1367] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:34.116835 2014 tablet_bootstrap.cc:654] T ac6730a31bab4499a2c97f95ea217a09 P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.116838 2007 raft_consensus.cc:348] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:34.117007 2007 raft_consensus.cc:374] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.117055 2007 raft_consensus.cc:729] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.117208 2007 consensus_queue.cc:260] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:34.117502 2007 ts_tablet_manager.cc:1398] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.118453 2009 tablet_bootstrap.cc:492] T 37732050f7a74053827b657e521f9a92 P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:34.118561 2009 ts_tablet_manager.cc:1367] T 37732050f7a74053827b657e521f9a92 P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.002s
I20240501 07:59:34.119083 2009 raft_consensus.cc:348] T 37732050f7a74053827b657e521f9a92 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.119204 2009 raft_consensus.cc:374] T 37732050f7a74053827b657e521f9a92 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.119246 2009 raft_consensus.cc:729] T 37732050f7a74053827b657e521f9a92 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.119393 2009 consensus_queue.cc:260] T 37732050f7a74053827b657e521f9a92 P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.119688 2009 ts_tablet_manager.cc:1398] T 37732050f7a74053827b657e521f9a92 P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:34.119884 2009 tablet_bootstrap.cc:492] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:34.120868 2009 tablet_bootstrap.cc:654] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.122184 2014 tablet_bootstrap.cc:492] T ac6730a31bab4499a2c97f95ea217a09 P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:34.122296 2014 ts_tablet_manager.cc:1367] T ac6730a31bab4499a2c97f95ea217a09 P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:34.122354 2009 tablet_bootstrap.cc:492] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:34.122442 2009 ts_tablet_manager.cc:1367] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:59:34.122830 2014 raft_consensus.cc:348] T ac6730a31bab4499a2c97f95ea217a09 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.122931 2014 raft_consensus.cc:374] T ac6730a31bab4499a2c97f95ea217a09 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.122951 2014 raft_consensus.cc:729] T ac6730a31bab4499a2c97f95ea217a09 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.123036 2009 raft_consensus.cc:348] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:34.123083 2014 consensus_queue.cc:260] T ac6730a31bab4499a2c97f95ea217a09 P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.123140 2009 raft_consensus.cc:374] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.123168 2009 raft_consensus.cc:729] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.123276 2009 consensus_queue.cc:260] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:34.123342 2014 ts_tablet_manager.cc:1398] T ac6730a31bab4499a2c97f95ea217a09 P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:34.123544 2009 ts_tablet_manager.cc:1398] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:34.156724 1938 raft_consensus.cc:482] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:34.156867 1938 raft_consensus.cc:504] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.157275 1938 leader_election.cc:288] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663), 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765)
I20240501 07:59:34.157701 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6730a31bab4499a2c97f95ea217a09" candidate_uuid: "c45c306768564fb2bd741c535d334429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1526c5008f714bcfa69e9bacb51a662c" is_pre_election: true
I20240501 07:59:34.157749 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6730a31bab4499a2c97f95ea217a09" candidate_uuid: "c45c306768564fb2bd741c535d334429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725" is_pre_election: true
I20240501 07:59:34.157866 979 raft_consensus.cc:2454] T ac6730a31bab4499a2c97f95ea217a09 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c45c306768564fb2bd741c535d334429 in term 0.
I20240501 07:59:34.157913 715 raft_consensus.cc:2454] T ac6730a31bab4499a2c97f95ea217a09 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c45c306768564fb2bd741c535d334429 in term 0.
I20240501 07:59:34.158179 1177 leader_election.cc:302] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:34.158346 1938 raft_consensus.cc:2781] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:34.158404 1938 raft_consensus.cc:482] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:34.158432 1938 raft_consensus.cc:3037] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.159255 1938 raft_consensus.cc:504] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.159580 1938 leader_election.cc:288] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [CANDIDATE]: Term 1 election: Requested vote from peers 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663), 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765)
I20240501 07:59:34.159926 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6730a31bab4499a2c97f95ea217a09" candidate_uuid: "c45c306768564fb2bd741c535d334429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1526c5008f714bcfa69e9bacb51a662c"
I20240501 07:59:34.159983 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6730a31bab4499a2c97f95ea217a09" candidate_uuid: "c45c306768564fb2bd741c535d334429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725"
I20240501 07:59:34.160039 979 raft_consensus.cc:3037] T ac6730a31bab4499a2c97f95ea217a09 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.160104 715 raft_consensus.cc:3037] T ac6730a31bab4499a2c97f95ea217a09 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.160825 715 raft_consensus.cc:2454] T ac6730a31bab4499a2c97f95ea217a09 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c45c306768564fb2bd741c535d334429 in term 1.
I20240501 07:59:34.160830 979 raft_consensus.cc:2454] T ac6730a31bab4499a2c97f95ea217a09 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c45c306768564fb2bd741c535d334429 in term 1.
I20240501 07:59:34.161103 1177 leader_election.cc:302] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee011df676c43f7beaeace0f8c60725, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:34.161257 1938 raft_consensus.cc:2781] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:34.161343 1938 raft_consensus.cc:686] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [term 1 LEADER]: Becoming Leader. State: Replica: c45c306768564fb2bd741c535d334429, State: Running, Role: LEADER
I20240501 07:59:34.161481 1938 consensus_queue.cc:237] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.163455 441 catalog_manager.cc:5484] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 reported cstate change: term changed from 0 to 1, leader changed from <none> to c45c306768564fb2bd741c535d334429 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "c45c306768564fb2bd741c535d334429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:34.175081 1654 raft_consensus.cc:482] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:34.175218 1654 raft_consensus.cc:504] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.175701 1654 leader_election.cc:288] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee66e185465431389043d933cb51308 (127.0.99.131:37541), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:34.176156 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37732050f7a74053827b657e521f9a92" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308" is_pre_election: true
I20240501 07:59:34.176338 847 raft_consensus.cc:2454] T 37732050f7a74053827b657e521f9a92 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 0.
I20240501 07:59:34.176652 518 leader_election.cc:302] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, b9644aeae9de4deba0c3a2b85df04747; no voters:
I20240501 07:59:34.176833 1654 raft_consensus.cc:2781] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:34.176884 1654 raft_consensus.cc:482] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:34.176928 1654 raft_consensus.cc:3037] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.177785 1654 raft_consensus.cc:504] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.178165 1654 leader_election.cc:288] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee66e185465431389043d933cb51308 (127.0.99.131:37541), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:34.178458 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37732050f7a74053827b657e521f9a92" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308"
I20240501 07:59:34.178601 847 raft_consensus.cc:3037] T 37732050f7a74053827b657e521f9a92 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.179380 847 raft_consensus.cc:2454] T 37732050f7a74053827b657e521f9a92 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 1.
I20240501 07:59:34.179800 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37732050f7a74053827b657e521f9a92" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec" is_pre_election: true
I20240501 07:59:34.179862 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37732050f7a74053827b657e521f9a92" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec"
I20240501 07:59:34.179885 518 leader_election.cc:302] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, b9644aeae9de4deba0c3a2b85df04747; no voters:
I20240501 07:59:34.179962 1109 raft_consensus.cc:3037] T 37732050f7a74053827b657e521f9a92 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.180053 1654 raft_consensus.cc:2781] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:34.180259 1654 raft_consensus.cc:686] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [term 1 LEADER]: Becoming Leader. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Running, Role: LEADER
I20240501 07:59:34.180413 1654 consensus_queue.cc:237] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.180605 1977 raft_consensus.cc:482] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:34.180703 1977 raft_consensus.cc:504] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:34.181018 1109 raft_consensus.cc:2454] T 37732050f7a74053827b657e521f9a92 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 1.
I20240501 07:59:34.181118 1977 leader_election.cc:288] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665), b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609)
I20240501 07:59:34.181535 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5c68ac4b29424e8f0dbb50dbdfb77f" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747" is_pre_election: true
I20240501 07:59:34.181664 582 raft_consensus.cc:2454] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:34.181943 781 leader_election.cc:302] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, b9644aeae9de4deba0c3a2b85df04747; no voters:
I20240501 07:59:34.182067 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5c68ac4b29424e8f0dbb50dbdfb77f" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec" is_pre_election: true
I20240501 07:59:34.182137 1977 raft_consensus.cc:2781] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:34.182184 1977 raft_consensus.cc:482] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:34.182201 1109 raft_consensus.cc:2454] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:34.182237 1977 raft_consensus.cc:3037] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.183161 1977 raft_consensus.cc:504] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:34.183107 441 catalog_manager.cc:5484] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b9644aeae9de4deba0c3a2b85df04747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:34.183503 1977 leader_election.cc:288] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Requested vote from peers f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665), b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609)
I20240501 07:59:34.183756 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5c68ac4b29424e8f0dbb50dbdfb77f" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec"
I20240501 07:59:34.183868 1109 raft_consensus.cc:3037] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.183993 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5c68ac4b29424e8f0dbb50dbdfb77f" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747"
I20240501 07:59:34.184084 582 raft_consensus.cc:3037] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.184528 1109 raft_consensus.cc:2454] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:34.184823 780 leader_election.cc:302] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:34.184962 1977 raft_consensus.cc:2781] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:34.185050 1977 raft_consensus.cc:686] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee66e185465431389043d933cb51308, State: Running, Role: LEADER
I20240501 07:59:34.185153 1977 consensus_queue.cc:237] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:34.186830 441 catalog_manager.cc:5484] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee66e185465431389043d933cb51308 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8ee66e185465431389043d933cb51308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:34.202853 582 raft_consensus.cc:2454] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
W20240501 07:59:34.307090 2028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:34.307394 2028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:34.310873 2028 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 07:59:34.425376 847 raft_consensus.cc:1261] T 37732050f7a74053827b657e521f9a92 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Refusing update from remote peer b9644aeae9de4deba0c3a2b85df04747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.425544 1109 raft_consensus.cc:1261] T 37732050f7a74053827b657e521f9a92 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Refusing update from remote peer b9644aeae9de4deba0c3a2b85df04747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.426234 2025 consensus_queue.cc:1035] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.426435 1654 consensus_queue.cc:1035] T 37732050f7a74053827b657e521f9a92 P b9644aeae9de4deba0c3a2b85df04747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.426671 715 raft_consensus.cc:1261] T ac6730a31bab4499a2c97f95ea217a09 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Refusing update from remote peer c45c306768564fb2bd741c535d334429: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.427191 1938 consensus_queue.cc:1035] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.427652 582 raft_consensus.cc:1261] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Refusing update from remote peer 8ee66e185465431389043d933cb51308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.428011 1977 consensus_queue.cc:1035] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.428390 1106 raft_consensus.cc:1261] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Refusing update from remote peer 8ee66e185465431389043d933cb51308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.428767 1977 consensus_queue.cc:1035] T 8d5c68ac4b29424e8f0dbb50dbdfb77f P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.454453 979 raft_consensus.cc:1261] T ac6730a31bab4499a2c97f95ea217a09 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Refusing update from remote peer c45c306768564fb2bd741c535d334429: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:34.455682 1520 consensus_queue.cc:1035] T ac6730a31bab4499a2c97f95ea217a09 P c45c306768564fb2bd741c535d334429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.461652 1975 mvcc.cc:205] Tried to move back new op lower bound from 7022798333642723328 to 7022798332567605248. Current Snapshot: MvccSnapshot[applied={T|T < 7022798333642723328}]
INSERT report
rows total: 2000
time total: 145.348 ms
time per row: 0.0726739 ms
I20240501 07:59:34.507068 436 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36176:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:34.513021 959 tablet_service.cc:1467] Processing CreateTablet for tablet 8e85dcd82f4e462ba9d1e8965a5938e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=190ca68e8bad49b49c3b03e876e058fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.513374 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e85dcd82f4e462ba9d1e8965a5938e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.514003 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 57adc45a29584ef388c671fd4e7c4cf8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=190ca68e8bad49b49c3b03e876e058fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.514297 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57adc45a29584ef388c671fd4e7c4cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.514521 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 8e85dcd82f4e462ba9d1e8965a5938e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=190ca68e8bad49b49c3b03e876e058fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.514802 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e85dcd82f4e462ba9d1e8965a5938e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.515058 694 tablet_service.cc:1467] Processing CreateTablet for tablet 8e85dcd82f4e462ba9d1e8965a5938e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=190ca68e8bad49b49c3b03e876e058fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.515358 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e85dcd82f4e462ba9d1e8965a5938e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.515733 695 tablet_service.cc:1467] Processing CreateTablet for tablet 699a8bf290e8422f97d4f828d03551eb (DEFAULT_TABLE table=rebalance_test_table_9 [id=190ca68e8bad49b49c3b03e876e058fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.515982 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 699a8bf290e8422f97d4f828d03551eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.516306 2013 tablet_bootstrap.cc:492] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:34.516852 2016 tablet_bootstrap.cc:492] T 8e85dcd82f4e462ba9d1e8965a5938e1 P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:34.517045 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 699a8bf290e8422f97d4f828d03551eb (DEFAULT_TABLE table=rebalance_test_table_9 [id=190ca68e8bad49b49c3b03e876e058fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.517189 2013 tablet_bootstrap.cc:654] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.517285 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 699a8bf290e8422f97d4f828d03551eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.517493 2009 tablet_bootstrap.cc:492] T 57adc45a29584ef388c671fd4e7c4cf8 P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:34.517766 2016 tablet_bootstrap.cc:654] T 8e85dcd82f4e462ba9d1e8965a5938e1 P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.517819 2014 tablet_bootstrap.cc:492] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:34.518411 2009 tablet_bootstrap.cc:654] T 57adc45a29584ef388c671fd4e7c4cf8 P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.518958 2014 tablet_bootstrap.cc:654] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.519445 2009 tablet_bootstrap.cc:492] T 57adc45a29584ef388c671fd4e7c4cf8 P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:34.519522 2009 ts_tablet_manager.cc:1367] T 57adc45a29584ef388c671fd4e7c4cf8 P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:34.519594 563 tablet_service.cc:1467] Processing CreateTablet for tablet 57adc45a29584ef388c671fd4e7c4cf8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=190ca68e8bad49b49c3b03e876e058fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.519873 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57adc45a29584ef388c671fd4e7c4cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.519980 2013 tablet_bootstrap.cc:492] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:34.520053 2013 ts_tablet_manager.cc:1367] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:34.519987 2009 raft_consensus.cc:348] T 57adc45a29584ef388c671fd4e7c4cf8 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.520112 2009 raft_consensus.cc:374] T 57adc45a29584ef388c671fd4e7c4cf8 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.520151 2009 raft_consensus.cc:729] T 57adc45a29584ef388c671fd4e7c4cf8 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.520272 2009 consensus_queue.cc:260] T 57adc45a29584ef388c671fd4e7c4cf8 P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.520474 2013 raft_consensus.cc:348] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.520541 2009 ts_tablet_manager.cc:1398] T 57adc45a29584ef388c671fd4e7c4cf8 P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.520568 2013 raft_consensus.cc:374] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.520598 2013 raft_consensus.cc:729] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.520692 2013 consensus_queue.cc:260] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.520977 2013 ts_tablet_manager.cc:1398] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.521649 826 tablet_service.cc:1467] Processing CreateTablet for tablet 57adc45a29584ef388c671fd4e7c4cf8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=190ca68e8bad49b49c3b03e876e058fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.521934 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57adc45a29584ef388c671fd4e7c4cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.522189 2008 tablet_bootstrap.cc:492] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:34.522521 2016 tablet_bootstrap.cc:492] T 8e85dcd82f4e462ba9d1e8965a5938e1 P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:34.522586 2016 ts_tablet_manager.cc:1367] T 8e85dcd82f4e462ba9d1e8965a5938e1 P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 07:59:34.522933 2016 raft_consensus.cc:348] T 8e85dcd82f4e462ba9d1e8965a5938e1 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.523026 2008 tablet_bootstrap.cc:654] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.523036 2016 raft_consensus.cc:374] T 8e85dcd82f4e462ba9d1e8965a5938e1 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.523066 2016 raft_consensus.cc:729] T 8e85dcd82f4e462ba9d1e8965a5938e1 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.523180 2016 consensus_queue.cc:260] T 8e85dcd82f4e462ba9d1e8965a5938e1 P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.523442 2016 ts_tablet_manager.cc:1398] T 8e85dcd82f4e462ba9d1e8965a5938e1 P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:34.523715 2016 tablet_bootstrap.cc:492] T 699a8bf290e8422f97d4f828d03551eb P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:34.524629 2016 tablet_bootstrap.cc:654] T 699a8bf290e8422f97d4f828d03551eb P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.525557 959 tablet_service.cc:1467] Processing CreateTablet for tablet 699a8bf290e8422f97d4f828d03551eb (DEFAULT_TABLE table=rebalance_test_table_9 [id=190ca68e8bad49b49c3b03e876e058fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.525840 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 699a8bf290e8422f97d4f828d03551eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.526122 2007 tablet_bootstrap.cc:492] T 57adc45a29584ef388c671fd4e7c4cf8 P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:34.526243 2008 tablet_bootstrap.cc:492] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:34.526324 2008 ts_tablet_manager.cc:1367] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:59:34.527033 2007 tablet_bootstrap.cc:654] T 57adc45a29584ef388c671fd4e7c4cf8 P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.527040 2008 raft_consensus.cc:348] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.527145 2008 raft_consensus.cc:374] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.527179 2008 raft_consensus.cc:729] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.527316 2008 consensus_queue.cc:260] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.527639 2008 ts_tablet_manager.cc:1398] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.529901 2007 tablet_bootstrap.cc:492] T 57adc45a29584ef388c671fd4e7c4cf8 P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:34.530032 2007 ts_tablet_manager.cc:1367] T 57adc45a29584ef388c671fd4e7c4cf8 P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:34.530347 2013 tablet_bootstrap.cc:492] T 699a8bf290e8422f97d4f828d03551eb P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:34.530519 2007 raft_consensus.cc:348] T 57adc45a29584ef388c671fd4e7c4cf8 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.530617 2007 raft_consensus.cc:374] T 57adc45a29584ef388c671fd4e7c4cf8 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.530647 2007 raft_consensus.cc:729] T 57adc45a29584ef388c671fd4e7c4cf8 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.530777 2007 consensus_queue.cc:260] T 57adc45a29584ef388c671fd4e7c4cf8 P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.531056 2007 ts_tablet_manager.cc:1398] T 57adc45a29584ef388c671fd4e7c4cf8 P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.531406 2013 tablet_bootstrap.cc:654] T 699a8bf290e8422f97d4f828d03551eb P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.532608 2014 tablet_bootstrap.cc:492] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:34.532698 2014 ts_tablet_manager.cc:1367] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 07:59:34.533169 2014 raft_consensus.cc:348] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.533298 2014 raft_consensus.cc:374] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.533331 2014 raft_consensus.cc:729] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.533478 2014 consensus_queue.cc:260] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.533779 2014 ts_tablet_manager.cc:1398] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.534018 2014 tablet_bootstrap.cc:492] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:34.534924 2014 tablet_bootstrap.cc:654] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.536247 2016 tablet_bootstrap.cc:492] T 699a8bf290e8422f97d4f828d03551eb P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:34.536255 2014 tablet_bootstrap.cc:492] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:34.536337 2014 ts_tablet_manager.cc:1367] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:34.536348 2016 ts_tablet_manager.cc:1367] T 699a8bf290e8422f97d4f828d03551eb P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20240501 07:59:34.536808 2014 raft_consensus.cc:348] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:34.536933 2014 raft_consensus.cc:374] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.536856 2016 raft_consensus.cc:348] T 699a8bf290e8422f97d4f828d03551eb P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:34.536967 2014 raft_consensus.cc:729] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.536994 2016 raft_consensus.cc:374] T 699a8bf290e8422f97d4f828d03551eb P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.537030 2016 raft_consensus.cc:729] T 699a8bf290e8422f97d4f828d03551eb P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.537094 2014 consensus_queue.cc:260] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:34.537154 2016 consensus_queue.cc:260] T 699a8bf290e8422f97d4f828d03551eb P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:34.537371 2014 ts_tablet_manager.cc:1398] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.538048 2016 ts_tablet_manager.cc:1398] T 699a8bf290e8422f97d4f828d03551eb P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:59:34.550484 2013 tablet_bootstrap.cc:492] T 699a8bf290e8422f97d4f828d03551eb P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:34.550622 2013 ts_tablet_manager.cc:1367] T 699a8bf290e8422f97d4f828d03551eb P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 07:59:34.551090 2013 raft_consensus.cc:348] T 699a8bf290e8422f97d4f828d03551eb P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:34.551223 2013 raft_consensus.cc:374] T 699a8bf290e8422f97d4f828d03551eb P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.551259 2013 raft_consensus.cc:729] T 699a8bf290e8422f97d4f828d03551eb P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.551405 2013 consensus_queue.cc:260] T 699a8bf290e8422f97d4f828d03551eb P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:34.551702 2013 ts_tablet_manager.cc:1398] T 699a8bf290e8422f97d4f828d03551eb P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.555968 2050 raft_consensus.cc:482] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:34.556077 2050 raft_consensus.cc:504] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.556444 2050 leader_election.cc:288] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c45c306768564fb2bd741c535d334429 (127.0.99.134:37067), 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765)
I20240501 07:59:34.556882 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e85dcd82f4e462ba9d1e8965a5938e1" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429" is_pre_election: true
I20240501 07:59:34.556885 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e85dcd82f4e462ba9d1e8965a5938e1" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725" is_pre_election: true
I20240501 07:59:34.557013 1243 raft_consensus.cc:2454] T 8e85dcd82f4e462ba9d1e8965a5938e1 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:34.557032 715 raft_consensus.cc:2454] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:34.557320 912 leader_election.cc:302] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:34.557471 2050 raft_consensus.cc:2781] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:34.557528 2050 raft_consensus.cc:482] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:34.557555 2050 raft_consensus.cc:3037] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.558468 2050 raft_consensus.cc:504] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.558817 2050 leader_election.cc:288] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Requested vote from peers c45c306768564fb2bd741c535d334429 (127.0.99.134:37067), 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765)
I20240501 07:59:34.559175 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e85dcd82f4e462ba9d1e8965a5938e1" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429"
I20240501 07:59:34.559296 1243 raft_consensus.cc:3037] T 8e85dcd82f4e462ba9d1e8965a5938e1 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.560226 1243 raft_consensus.cc:2454] T 8e85dcd82f4e462ba9d1e8965a5938e1 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:34.560467 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e85dcd82f4e462ba9d1e8965a5938e1" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725"
I20240501 07:59:34.560530 912 leader_election.cc:302] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:34.560608 715 raft_consensus.cc:3037] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.560771 2050 raft_consensus.cc:2781] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:34.560863 2050 raft_consensus.cc:686] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [term 1 LEADER]: Becoming Leader. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Running, Role: LEADER
I20240501 07:59:34.561009 2050 consensus_queue.cc:237] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } }
I20240501 07:59:34.561640 715 raft_consensus.cc:2454] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:34.562922 439 catalog_manager.cc:5484] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1526c5008f714bcfa69e9bacb51a662c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:34.627585 1940 raft_consensus.cc:482] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:34.627732 1940 raft_consensus.cc:504] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:34.628212 1940 leader_election.cc:288] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:34.628633 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a8bf290e8422f97d4f828d03551eb" candidate_uuid: "3ee011df676c43f7beaeace0f8c60725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1526c5008f714bcfa69e9bacb51a662c" is_pre_election: true
I20240501 07:59:34.628692 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a8bf290e8422f97d4f828d03551eb" candidate_uuid: "3ee011df676c43f7beaeace0f8c60725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429" is_pre_election: true
I20240501 07:59:34.628814 1243 raft_consensus.cc:2454] T 699a8bf290e8422f97d4f828d03551eb P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ee011df676c43f7beaeace0f8c60725 in term 0.
I20240501 07:59:34.628815 979 raft_consensus.cc:2454] T 699a8bf290e8422f97d4f828d03551eb P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ee011df676c43f7beaeace0f8c60725 in term 0.
I20240501 07:59:34.629118 648 leader_election.cc:302] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee011df676c43f7beaeace0f8c60725, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:34.629348 1940 raft_consensus.cc:2781] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:34.629420 1940 raft_consensus.cc:482] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:34.629472 1940 raft_consensus.cc:3037] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.630417 1940 raft_consensus.cc:504] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:34.630775 1940 leader_election.cc:288] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [CANDIDATE]: Term 1 election: Requested vote from peers 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:34.631125 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a8bf290e8422f97d4f828d03551eb" candidate_uuid: "3ee011df676c43f7beaeace0f8c60725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429"
I20240501 07:59:34.631246 1243 raft_consensus.cc:3037] T 699a8bf290e8422f97d4f828d03551eb P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.631204 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a8bf290e8422f97d4f828d03551eb" candidate_uuid: "3ee011df676c43f7beaeace0f8c60725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1526c5008f714bcfa69e9bacb51a662c"
I20240501 07:59:34.631335 979 raft_consensus.cc:3037] T 699a8bf290e8422f97d4f828d03551eb P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.632115 979 raft_consensus.cc:2454] T 699a8bf290e8422f97d4f828d03551eb P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ee011df676c43f7beaeace0f8c60725 in term 1.
I20240501 07:59:34.632203 1243 raft_consensus.cc:2454] T 699a8bf290e8422f97d4f828d03551eb P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ee011df676c43f7beaeace0f8c60725 in term 1.
I20240501 07:59:34.632509 648 leader_election.cc:302] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee011df676c43f7beaeace0f8c60725, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:34.632688 1940 raft_consensus.cc:2781] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:34.632858 1940 raft_consensus.cc:686] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [term 1 LEADER]: Becoming Leader. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Running, Role: LEADER
I20240501 07:59:34.633036 1940 consensus_queue.cc:237] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:34.634747 439 catalog_manager.cc:5484] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3ee011df676c43f7beaeace0f8c60725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:34.834120 1654 raft_consensus.cc:482] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:34.834245 1654 raft_consensus.cc:504] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.834640 1654 leader_election.cc:288] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee66e185465431389043d933cb51308 (127.0.99.131:37541), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:34.835091 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57adc45a29584ef388c671fd4e7c4cf8" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308" is_pre_election: true
I20240501 07:59:34.835098 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57adc45a29584ef388c671fd4e7c4cf8" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec" is_pre_election: true
I20240501 07:59:34.835245 847 raft_consensus.cc:2454] T 57adc45a29584ef388c671fd4e7c4cf8 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 0.
I20240501 07:59:34.835249 1106 raft_consensus.cc:2454] T 57adc45a29584ef388c671fd4e7c4cf8 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 0.
I20240501 07:59:34.835541 516 leader_election.cc:302] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9644aeae9de4deba0c3a2b85df04747, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:34.835736 1654 raft_consensus.cc:2781] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:34.835812 1654 raft_consensus.cc:482] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:34.835862 1654 raft_consensus.cc:3037] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.836705 1654 raft_consensus.cc:504] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.837050 1654 leader_election.cc:288] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee66e185465431389043d933cb51308 (127.0.99.131:37541), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:34.837390 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57adc45a29584ef388c671fd4e7c4cf8" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308"
I20240501 07:59:34.837437 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57adc45a29584ef388c671fd4e7c4cf8" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec"
I20240501 07:59:34.837487 847 raft_consensus.cc:3037] T 57adc45a29584ef388c671fd4e7c4cf8 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.837563 1106 raft_consensus.cc:3037] T 57adc45a29584ef388c671fd4e7c4cf8 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.838380 1106 raft_consensus.cc:2454] T 57adc45a29584ef388c671fd4e7c4cf8 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 1.
I20240501 07:59:34.838413 847 raft_consensus.cc:2454] T 57adc45a29584ef388c671fd4e7c4cf8 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 1.
I20240501 07:59:34.838680 516 leader_election.cc:302] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9644aeae9de4deba0c3a2b85df04747, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:34.838840 1654 raft_consensus.cc:2781] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:34.838925 1654 raft_consensus.cc:686] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [term 1 LEADER]: Becoming Leader. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Running, Role: LEADER
I20240501 07:59:34.839066 1654 consensus_queue.cc:237] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:34.841121 439 catalog_manager.cc:5484] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b9644aeae9de4deba0c3a2b85df04747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:34.975724 2050 consensus_queue.cc:1035] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.985267 1493 consensus_queue.cc:1035] T 8e85dcd82f4e462ba9d1e8965a5938e1 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:59:35.009295 2082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:35.009635 2082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:35.013229 2082 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 07:59:35.047415 2078 consensus_queue.cc:1035] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.050829 1940 consensus_queue.cc:1035] T 699a8bf290e8422f97d4f828d03551eb P 3ee011df676c43f7beaeace0f8c60725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.139710 1106 raft_consensus.cc:1261] T 57adc45a29584ef388c671fd4e7c4cf8 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Refusing update from remote peer b9644aeae9de4deba0c3a2b85df04747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.140667 1654 consensus_queue.cc:1035] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.144402 846 raft_consensus.cc:1261] T 57adc45a29584ef388c671fd4e7c4cf8 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Refusing update from remote peer b9644aeae9de4deba0c3a2b85df04747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.145087 1654 consensus_queue.cc:1035] T 57adc45a29584ef388c671fd4e7c4cf8 P b9644aeae9de4deba0c3a2b85df04747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 166.77 ms
time per row: 0.0833848 ms
I20240501 07:59:35.234701 439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36176:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:35.242061 563 tablet_service.cc:1467] Processing CreateTablet for tablet 5a864679fa6948f5862417722e296302 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2c48bb67d61f4c0cb8c4d22e32c4c718]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.242421 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a864679fa6948f5862417722e296302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.242534 959 tablet_service.cc:1467] Processing CreateTablet for tablet 5838f08a8fd04e2a9fcf192684503bf3 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2c48bb67d61f4c0cb8c4d22e32c4c718]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.242790 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5838f08a8fd04e2a9fcf192684503bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.245157 2151 tablet_bootstrap.cc:492] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:35.245294 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 5a864679fa6948f5862417722e296302 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2c48bb67d61f4c0cb8c4d22e32c4c718]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.245579 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a864679fa6948f5862417722e296302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.245857 695 tablet_service.cc:1467] Processing CreateTablet for tablet 5838f08a8fd04e2a9fcf192684503bf3 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2c48bb67d61f4c0cb8c4d22e32c4c718]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.246168 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5838f08a8fd04e2a9fcf192684503bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.246218 2151 tablet_bootstrap.cc:654] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.247387 2151 tablet_bootstrap.cc:492] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:35.247462 2151 ts_tablet_manager.cc:1367] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:35.247659 2153 tablet_bootstrap.cc:492] T 5a864679fa6948f5862417722e296302 P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:35.247893 2151 raft_consensus.cc:348] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.247921 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 5838f08a8fd04e2a9fcf192684503bf3 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2c48bb67d61f4c0cb8c4d22e32c4c718]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.248008 2151 raft_consensus.cc:374] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.248040 2151 raft_consensus.cc:729] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.248188 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5838f08a8fd04e2a9fcf192684503bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.248171 2151 consensus_queue.cc:260] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.248797 2153 tablet_bootstrap.cc:654] T 5a864679fa6948f5862417722e296302 P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.249294 2151 ts_tablet_manager.cc:1398] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:35.249593 2154 tablet_bootstrap.cc:492] T 5838f08a8fd04e2a9fcf192684503bf3 P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:35.250559 826 tablet_service.cc:1467] Processing CreateTablet for tablet 5a864679fa6948f5862417722e296302 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2c48bb67d61f4c0cb8c4d22e32c4c718]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.250766 2154 tablet_bootstrap.cc:654] T 5838f08a8fd04e2a9fcf192684503bf3 P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.250824 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a864679fa6948f5862417722e296302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.251518 827 tablet_service.cc:1467] Processing CreateTablet for tablet 73ba1454ed0d4be0bf354a07c5fc7980 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2c48bb67d61f4c0cb8c4d22e32c4c718]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.251754 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ba1454ed0d4be0bf354a07c5fc7980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.252245 2154 tablet_bootstrap.cc:492] T 5838f08a8fd04e2a9fcf192684503bf3 P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:35.252342 2154 ts_tablet_manager.cc:1367] T 5838f08a8fd04e2a9fcf192684503bf3 P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:35.252961 2154 raft_consensus.cc:348] T 5838f08a8fd04e2a9fcf192684503bf3 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.253080 2154 raft_consensus.cc:374] T 5838f08a8fd04e2a9fcf192684503bf3 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.253121 2154 raft_consensus.cc:729] T 5838f08a8fd04e2a9fcf192684503bf3 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.253285 2154 consensus_queue.cc:260] T 5838f08a8fd04e2a9fcf192684503bf3 P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.253690 2154 ts_tablet_manager.cc:1398] T 5838f08a8fd04e2a9fcf192684503bf3 P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:35.255671 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 73ba1454ed0d4be0bf354a07c5fc7980 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2c48bb67d61f4c0cb8c4d22e32c4c718]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.255967 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ba1454ed0d4be0bf354a07c5fc7980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.258349 563 tablet_service.cc:1467] Processing CreateTablet for tablet 73ba1454ed0d4be0bf354a07c5fc7980 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2c48bb67d61f4c0cb8c4d22e32c4c718]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.258620 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ba1454ed0d4be0bf354a07c5fc7980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.260489 2150 tablet_bootstrap.cc:492] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:35.261251 2150 tablet_bootstrap.cc:654] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.262207 2156 tablet_bootstrap.cc:492] T 5a864679fa6948f5862417722e296302 P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:35.262481 2150 tablet_bootstrap.cc:492] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:35.263160 2156 tablet_bootstrap.cc:654] T 5a864679fa6948f5862417722e296302 P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.261952 2157 tablet_bootstrap.cc:492] T 5838f08a8fd04e2a9fcf192684503bf3 P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:35.265228 2157 tablet_bootstrap.cc:654] T 5838f08a8fd04e2a9fcf192684503bf3 P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.265964 2150 ts_tablet_manager.cc:1367] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:35.266950 2150 raft_consensus.cc:348] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:35.267052 2153 tablet_bootstrap.cc:492] T 5a864679fa6948f5862417722e296302 P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:35.267153 2153 ts_tablet_manager.cc:1367] T 5a864679fa6948f5862417722e296302 P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 07:59:35.267166 2150 raft_consensus.cc:374] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.267294 2150 raft_consensus.cc:729] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.267505 2150 consensus_queue.cc:260] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:35.267654 2153 raft_consensus.cc:348] T 5a864679fa6948f5862417722e296302 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:35.267764 2153 raft_consensus.cc:374] T 5a864679fa6948f5862417722e296302 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.267797 2153 raft_consensus.cc:729] T 5a864679fa6948f5862417722e296302 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.267930 2153 consensus_queue.cc:260] T 5a864679fa6948f5862417722e296302 P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:35.268221 2153 ts_tablet_manager.cc:1398] T 5a864679fa6948f5862417722e296302 P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:35.268406 2153 tablet_bootstrap.cc:492] T 73ba1454ed0d4be0bf354a07c5fc7980 P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:35.269387 2153 tablet_bootstrap.cc:654] T 73ba1454ed0d4be0bf354a07c5fc7980 P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.271559 2150 ts_tablet_manager.cc:1398] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 07:59:35.272384 2150 tablet_bootstrap.cc:492] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:35.275548 2157 tablet_bootstrap.cc:492] T 5838f08a8fd04e2a9fcf192684503bf3 P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:35.275954 2157 ts_tablet_manager.cc:1367] T 5838f08a8fd04e2a9fcf192684503bf3 P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20240501 07:59:35.276715 2150 tablet_bootstrap.cc:654] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.277211 2153 tablet_bootstrap.cc:492] T 73ba1454ed0d4be0bf354a07c5fc7980 P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:35.277129 2157 raft_consensus.cc:348] T 5838f08a8fd04e2a9fcf192684503bf3 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.277503 2153 ts_tablet_manager.cc:1367] T 73ba1454ed0d4be0bf354a07c5fc7980 P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 07:59:35.277498 2157 raft_consensus.cc:374] T 5838f08a8fd04e2a9fcf192684503bf3 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.277586 2157 raft_consensus.cc:729] T 5838f08a8fd04e2a9fcf192684503bf3 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.277894 2157 consensus_queue.cc:260] T 5838f08a8fd04e2a9fcf192684503bf3 P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.278296 2157 ts_tablet_manager.cc:1398] T 5838f08a8fd04e2a9fcf192684503bf3 P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:35.278342 2153 raft_consensus.cc:348] T 73ba1454ed0d4be0bf354a07c5fc7980 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.278457 2153 raft_consensus.cc:374] T 73ba1454ed0d4be0bf354a07c5fc7980 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.278497 2153 raft_consensus.cc:729] T 73ba1454ed0d4be0bf354a07c5fc7980 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.278656 2153 consensus_queue.cc:260] T 73ba1454ed0d4be0bf354a07c5fc7980 P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.279062 2153 ts_tablet_manager.cc:1398] T 73ba1454ed0d4be0bf354a07c5fc7980 P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:35.282670 2156 tablet_bootstrap.cc:492] T 5a864679fa6948f5862417722e296302 P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:35.282842 2156 ts_tablet_manager.cc:1367] T 5a864679fa6948f5862417722e296302 P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.002s
I20240501 07:59:35.283305 2156 raft_consensus.cc:348] T 5a864679fa6948f5862417722e296302 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:35.283416 2156 raft_consensus.cc:374] T 5a864679fa6948f5862417722e296302 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.283449 2156 raft_consensus.cc:729] T 5a864679fa6948f5862417722e296302 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.283632 2156 consensus_queue.cc:260] T 5a864679fa6948f5862417722e296302 P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:35.284010 2156 ts_tablet_manager.cc:1398] T 5a864679fa6948f5862417722e296302 P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:35.284265 2156 tablet_bootstrap.cc:492] T 73ba1454ed0d4be0bf354a07c5fc7980 P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:35.286674 2156 tablet_bootstrap.cc:654] T 73ba1454ed0d4be0bf354a07c5fc7980 P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.287979 2156 tablet_bootstrap.cc:492] T 73ba1454ed0d4be0bf354a07c5fc7980 P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:35.288077 2156 ts_tablet_manager.cc:1367] T 73ba1454ed0d4be0bf354a07c5fc7980 P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 07:59:35.288235 2150 tablet_bootstrap.cc:492] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:35.288341 2150 ts_tablet_manager.cc:1367] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20240501 07:59:35.288549 2156 raft_consensus.cc:348] T 73ba1454ed0d4be0bf354a07c5fc7980 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.288650 2156 raft_consensus.cc:374] T 73ba1454ed0d4be0bf354a07c5fc7980 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.288690 2156 raft_consensus.cc:729] T 73ba1454ed0d4be0bf354a07c5fc7980 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.288779 2150 raft_consensus.cc:348] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.288818 2156 consensus_queue.cc:260] T 73ba1454ed0d4be0bf354a07c5fc7980 P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.288885 2150 raft_consensus.cc:374] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.288915 2150 raft_consensus.cc:729] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.289032 2150 consensus_queue.cc:260] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.289296 2150 ts_tablet_manager.cc:1398] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:35.289312 2156 ts_tablet_manager.cc:1398] T 73ba1454ed0d4be0bf354a07c5fc7980 P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:35.293602 2135 raft_consensus.cc:482] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:35.293733 2135 raft_consensus.cc:504] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.294181 2135 leader_election.cc:288] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665), 8ee66e185465431389043d933cb51308 (127.0.99.131:37541)
I20240501 07:59:35.294513 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ba1454ed0d4be0bf354a07c5fc7980" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308" is_pre_election: true
I20240501 07:59:35.294526 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ba1454ed0d4be0bf354a07c5fc7980" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec" is_pre_election: true
I20240501 07:59:35.294639 846 raft_consensus.cc:2454] T 73ba1454ed0d4be0bf354a07c5fc7980 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 0.
I20240501 07:59:35.294675 1106 raft_consensus.cc:2454] T 73ba1454ed0d4be0bf354a07c5fc7980 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 0.
I20240501 07:59:35.294948 518 leader_election.cc:302] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, b9644aeae9de4deba0c3a2b85df04747; no voters:
I20240501 07:59:35.295173 2135 raft_consensus.cc:2781] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:35.295220 2135 raft_consensus.cc:482] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:35.295251 2135 raft_consensus.cc:3037] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.296154 2135 raft_consensus.cc:504] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.296532 2135 leader_election.cc:288] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 election: Requested vote from peers f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665), 8ee66e185465431389043d933cb51308 (127.0.99.131:37541)
I20240501 07:59:35.296909 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ba1454ed0d4be0bf354a07c5fc7980" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308"
I20240501 07:59:35.297036 846 raft_consensus.cc:3037] T 73ba1454ed0d4be0bf354a07c5fc7980 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.297053 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ba1454ed0d4be0bf354a07c5fc7980" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec"
I20240501 07:59:35.297165 1106 raft_consensus.cc:3037] T 73ba1454ed0d4be0bf354a07c5fc7980 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.297792 846 raft_consensus.cc:2454] T 73ba1454ed0d4be0bf354a07c5fc7980 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 1.
I20240501 07:59:35.297914 1106 raft_consensus.cc:2454] T 73ba1454ed0d4be0bf354a07c5fc7980 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 1.
I20240501 07:59:35.298170 518 leader_election.cc:302] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, b9644aeae9de4deba0c3a2b85df04747; no voters:
I20240501 07:59:35.298329 2135 raft_consensus.cc:2781] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:35.298432 2135 raft_consensus.cc:686] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [term 1 LEADER]: Becoming Leader. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Running, Role: LEADER
I20240501 07:59:35.298579 2135 consensus_queue.cc:237] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.300307 439 catalog_manager.cc:5484] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b9644aeae9de4deba0c3a2b85df04747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:35.322506 2050 raft_consensus.cc:482] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:35.322649 2050 raft_consensus.cc:504] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.323134 2050 leader_election.cc:288] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:35.323586 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5838f08a8fd04e2a9fcf192684503bf3" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429" is_pre_election: true
I20240501 07:59:35.323586 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5838f08a8fd04e2a9fcf192684503bf3" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725" is_pre_election: true
I20240501 07:59:35.323750 715 raft_consensus.cc:2454] T 5838f08a8fd04e2a9fcf192684503bf3 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:35.323750 1243 raft_consensus.cc:2454] T 5838f08a8fd04e2a9fcf192684503bf3 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:35.324142 912 leader_election.cc:302] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:35.324331 2050 raft_consensus.cc:2781] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:35.324406 2050 raft_consensus.cc:482] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:35.324456 2050 raft_consensus.cc:3037] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.325577 2135 raft_consensus.cc:482] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:35.325549 2050 raft_consensus.cc:504] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.325677 2135 raft_consensus.cc:504] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:35.325951 2050 leader_election.cc:288] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Requested vote from peers 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:35.326264 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5838f08a8fd04e2a9fcf192684503bf3" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429"
I20240501 07:59:35.326380 1243 raft_consensus.cc:3037] T 5838f08a8fd04e2a9fcf192684503bf3 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.326505 2135 leader_election.cc:288] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee66e185465431389043d933cb51308 (127.0.99.131:37541), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:35.326476 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5838f08a8fd04e2a9fcf192684503bf3" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725"
I20240501 07:59:35.326593 715 raft_consensus.cc:3037] T 5838f08a8fd04e2a9fcf192684503bf3 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.326911 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a864679fa6948f5862417722e296302" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec" is_pre_election: true
I20240501 07:59:35.327057 1106 raft_consensus.cc:2454] T 5a864679fa6948f5862417722e296302 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 0.
I20240501 07:59:35.327364 516 leader_election.cc:302] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9644aeae9de4deba0c3a2b85df04747, f5af98ef61b54ae0b33faeb78af96dec; no voters:
I20240501 07:59:35.327451 1243 raft_consensus.cc:2454] T 5838f08a8fd04e2a9fcf192684503bf3 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:35.327594 2135 raft_consensus.cc:2781] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:35.327663 2135 raft_consensus.cc:482] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:35.327709 2135 raft_consensus.cc:3037] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.327724 715 raft_consensus.cc:2454] T 5838f08a8fd04e2a9fcf192684503bf3 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:35.327714 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a864679fa6948f5862417722e296302" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308" is_pre_election: true
I20240501 07:59:35.327740 912 leader_election.cc:302] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:35.327849 846 raft_consensus.cc:2454] T 5a864679fa6948f5862417722e296302 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 0.
I20240501 07:59:35.328080 2050 raft_consensus.cc:2781] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:35.328178 2050 raft_consensus.cc:686] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [term 1 LEADER]: Becoming Leader. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Running, Role: LEADER
I20240501 07:59:35.328330 2050 consensus_queue.cc:237] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.329015 2135 raft_consensus.cc:504] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:35.329758 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a864679fa6948f5862417722e296302" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee66e185465431389043d933cb51308"
I20240501 07:59:35.329874 846 raft_consensus.cc:3037] T 5a864679fa6948f5862417722e296302 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.330821 846 raft_consensus.cc:2454] T 5a864679fa6948f5862417722e296302 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 1.
I20240501 07:59:35.330724 439 catalog_manager.cc:5484] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1526c5008f714bcfa69e9bacb51a662c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:35.331111 518 leader_election.cc:302] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, b9644aeae9de4deba0c3a2b85df04747; no voters:
I20240501 07:59:35.331295 1654 raft_consensus.cc:2781] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:35.331385 1654 raft_consensus.cc:686] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [term 1 LEADER]: Becoming Leader. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Running, Role: LEADER
I20240501 07:59:35.331511 1654 consensus_queue.cc:237] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } }
I20240501 07:59:35.331916 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a864679fa6948f5862417722e296302" candidate_uuid: "b9644aeae9de4deba0c3a2b85df04747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec"
I20240501 07:59:35.332026 1106 raft_consensus.cc:3037] T 5a864679fa6948f5862417722e296302 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.332973 1106 raft_consensus.cc:2454] T 5a864679fa6948f5862417722e296302 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9644aeae9de4deba0c3a2b85df04747 in term 1.
I20240501 07:59:35.333863 2135 leader_election.cc:288] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee66e185465431389043d933cb51308 (127.0.99.131:37541), f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665)
I20240501 07:59:35.334400 439 catalog_manager.cc:5484] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b9644aeae9de4deba0c3a2b85df04747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:35.461567 2168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:35.461815 2168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:35.465504 2168 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240501 07:59:35.557443 1106 raft_consensus.cc:1261] T 5a864679fa6948f5862417722e296302 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Refusing update from remote peer b9644aeae9de4deba0c3a2b85df04747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.557794 715 raft_consensus.cc:1261] T 5838f08a8fd04e2a9fcf192684503bf3 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Refusing update from remote peer 1526c5008f714bcfa69e9bacb51a662c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.558022 2194 consensus_queue.cc:1035] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.558571 1243 raft_consensus.cc:1261] T 5838f08a8fd04e2a9fcf192684503bf3 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Refusing update from remote peer 1526c5008f714bcfa69e9bacb51a662c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.558604 1109 raft_consensus.cc:1261] T 73ba1454ed0d4be0bf354a07c5fc7980 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Refusing update from remote peer b9644aeae9de4deba0c3a2b85df04747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.558637 2050 consensus_queue.cc:1035] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.559207 846 raft_consensus.cc:1261] T 73ba1454ed0d4be0bf354a07c5fc7980 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Refusing update from remote peer b9644aeae9de4deba0c3a2b85df04747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.559278 2050 consensus_queue.cc:1035] T 5838f08a8fd04e2a9fcf192684503bf3 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.559207 847 raft_consensus.cc:1261] T 5a864679fa6948f5862417722e296302 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Refusing update from remote peer b9644aeae9de4deba0c3a2b85df04747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.563297 2025 consensus_queue.cc:1035] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.563656 2025 consensus_queue.cc:1035] T 73ba1454ed0d4be0bf354a07c5fc7980 P b9644aeae9de4deba0c3a2b85df04747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.563925 2025 consensus_queue.cc:1035] T 5a864679fa6948f5862417722e296302 P b9644aeae9de4deba0c3a2b85df04747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.577416 2126 mvcc.cc:205] Tried to move back new op lower bound from 7022798338275782656 to 7022798337346338816. Current Snapshot: MvccSnapshot[applied={T|T < 7022798338275782656 or (T in {7022798338275782656})}]
I20240501 07:59:35.580240 2123 mvcc.cc:205] Tried to move back new op lower bound from 7022798338277281792 to 7022798337359335424. Current Snapshot: MvccSnapshot[applied={T|T < 7022798338277281792}]
I20240501 07:59:35.586750 2131 mvcc.cc:205] Tried to move back new op lower bound from 7022798338272501760 to 7022798337224404992. Current Snapshot: MvccSnapshot[applied={T|T < 7022798338272501760}]
INSERT report
rows total: 2000
time total: 112.098 ms
time per row: 0.0560491 ms
I20240501 07:59:35.627991 2128 mvcc.cc:205] Tried to move back new op lower bound from 7022798338277281792 to 7022798337359335424. Current Snapshot: MvccSnapshot[applied={T|T < 7022798338277281792}]
I20240501 07:59:35.632800 439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36176:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:35.639806 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 01402c21678c49ca9a143bb2272a549c (DEFAULT_TABLE table=rebalance_test_table_11 [id=915f3ebe6a75462c95ac14e9277c32ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.639886 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 596a573d62964e01b0726168616f7f08 (DEFAULT_TABLE table=rebalance_test_table_11 [id=915f3ebe6a75462c95ac14e9277c32ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.640153 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01402c21678c49ca9a143bb2272a549c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.640362 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596a573d62964e01b0726168616f7f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.641158 959 tablet_service.cc:1467] Processing CreateTablet for tablet 01402c21678c49ca9a143bb2272a549c (DEFAULT_TABLE table=rebalance_test_table_11 [id=915f3ebe6a75462c95ac14e9277c32ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.641463 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01402c21678c49ca9a143bb2272a549c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.642331 827 tablet_service.cc:1467] Processing CreateTablet for tablet 15d5127ae40944f0a7ac301047245472 (DEFAULT_TABLE table=rebalance_test_table_11 [id=915f3ebe6a75462c95ac14e9277c32ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.642481 694 tablet_service.cc:1467] Processing CreateTablet for tablet 596a573d62964e01b0726168616f7f08 (DEFAULT_TABLE table=rebalance_test_table_11 [id=915f3ebe6a75462c95ac14e9277c32ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.642597 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15d5127ae40944f0a7ac301047245472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.642724 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596a573d62964e01b0726168616f7f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.643568 2151 tablet_bootstrap.cc:492] T 01402c21678c49ca9a143bb2272a549c P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:35.643792 958 tablet_service.cc:1467] Processing CreateTablet for tablet 596a573d62964e01b0726168616f7f08 (DEFAULT_TABLE table=rebalance_test_table_11 [id=915f3ebe6a75462c95ac14e9277c32ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.644037 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596a573d62964e01b0726168616f7f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.644551 2151 tablet_bootstrap.cc:654] T 01402c21678c49ca9a143bb2272a549c P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.645020 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 15d5127ae40944f0a7ac301047245472 (DEFAULT_TABLE table=rebalance_test_table_11 [id=915f3ebe6a75462c95ac14e9277c32ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.642724 695 tablet_service.cc:1467] Processing CreateTablet for tablet 01402c21678c49ca9a143bb2272a549c (DEFAULT_TABLE table=rebalance_test_table_11 [id=915f3ebe6a75462c95ac14e9277c32ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.645287 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15d5127ae40944f0a7ac301047245472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.645437 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01402c21678c49ca9a143bb2272a549c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.646116 2157 tablet_bootstrap.cc:492] T 596a573d62964e01b0726168616f7f08 P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:35.646116 2154 tablet_bootstrap.cc:492] T 596a573d62964e01b0726168616f7f08 P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:35.647037 2154 tablet_bootstrap.cc:654] T 596a573d62964e01b0726168616f7f08 P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.647037 2157 tablet_bootstrap.cc:654] T 596a573d62964e01b0726168616f7f08 P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.647691 2156 tablet_bootstrap.cc:492] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308: Bootstrap starting.
I20240501 07:59:35.647866 563 tablet_service.cc:1467] Processing CreateTablet for tablet 15d5127ae40944f0a7ac301047245472 (DEFAULT_TABLE table=rebalance_test_table_11 [id=915f3ebe6a75462c95ac14e9277c32ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.648126 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15d5127ae40944f0a7ac301047245472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.648581 2156 tablet_bootstrap.cc:654] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.649649 2156 tablet_bootstrap.cc:492] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308: No bootstrap required, opened a new log
I20240501 07:59:35.649729 2156 ts_tablet_manager.cc:1367] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:35.650365 2156 raft_consensus.cc:348] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.650485 2156 raft_consensus.cc:374] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.650472 2150 tablet_bootstrap.cc:492] T 15d5127ae40944f0a7ac301047245472 P b9644aeae9de4deba0c3a2b85df04747: Bootstrap starting.
I20240501 07:59:35.650522 2156 raft_consensus.cc:729] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee66e185465431389043d933cb51308, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.650652 2156 consensus_queue.cc:260] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.650929 2153 tablet_bootstrap.cc:492] T 15d5127ae40944f0a7ac301047245472 P f5af98ef61b54ae0b33faeb78af96dec: Bootstrap starting.
I20240501 07:59:35.650971 2156 ts_tablet_manager.cc:1398] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:35.651203 2157 tablet_bootstrap.cc:492] T 596a573d62964e01b0726168616f7f08 P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:35.651304 2157 ts_tablet_manager.cc:1367] T 596a573d62964e01b0726168616f7f08 P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 07:59:35.651341 2150 tablet_bootstrap.cc:654] T 15d5127ae40944f0a7ac301047245472 P b9644aeae9de4deba0c3a2b85df04747: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.651787 2157 raft_consensus.cc:348] T 596a573d62964e01b0726168616f7f08 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:35.651911 2157 raft_consensus.cc:374] T 596a573d62964e01b0726168616f7f08 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.651921 2153 tablet_bootstrap.cc:654] T 15d5127ae40944f0a7ac301047245472 P f5af98ef61b54ae0b33faeb78af96dec: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.651945 2157 raft_consensus.cc:729] T 596a573d62964e01b0726168616f7f08 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.652074 2157 consensus_queue.cc:260] T 596a573d62964e01b0726168616f7f08 P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:35.652366 2157 ts_tablet_manager.cc:1398] T 596a573d62964e01b0726168616f7f08 P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:35.652540 2150 tablet_bootstrap.cc:492] T 15d5127ae40944f0a7ac301047245472 P b9644aeae9de4deba0c3a2b85df04747: No bootstrap required, opened a new log
I20240501 07:59:35.652616 2150 ts_tablet_manager.cc:1367] T 15d5127ae40944f0a7ac301047245472 P b9644aeae9de4deba0c3a2b85df04747: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:35.653039 2150 raft_consensus.cc:348] T 15d5127ae40944f0a7ac301047245472 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.653136 2150 raft_consensus.cc:374] T 15d5127ae40944f0a7ac301047245472 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.653179 2150 raft_consensus.cc:729] T 15d5127ae40944f0a7ac301047245472 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9644aeae9de4deba0c3a2b85df04747, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.653308 2150 consensus_queue.cc:260] T 15d5127ae40944f0a7ac301047245472 P b9644aeae9de4deba0c3a2b85df04747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.653577 2150 ts_tablet_manager.cc:1398] T 15d5127ae40944f0a7ac301047245472 P b9644aeae9de4deba0c3a2b85df04747: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:35.653728 2157 tablet_bootstrap.cc:492] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429: Bootstrap starting.
I20240501 07:59:35.654157 2154 tablet_bootstrap.cc:492] T 596a573d62964e01b0726168616f7f08 P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:35.654263 2154 ts_tablet_manager.cc:1367] T 596a573d62964e01b0726168616f7f08 P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:35.654758 2154 raft_consensus.cc:348] T 596a573d62964e01b0726168616f7f08 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:35.654875 2154 raft_consensus.cc:374] T 596a573d62964e01b0726168616f7f08 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.654912 2154 raft_consensus.cc:729] T 596a573d62964e01b0726168616f7f08 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.655022 2157 tablet_bootstrap.cc:654] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.655062 2154 consensus_queue.cc:260] T 596a573d62964e01b0726168616f7f08 P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:35.655387 2154 ts_tablet_manager.cc:1398] T 596a573d62964e01b0726168616f7f08 P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:35.655550 2154 tablet_bootstrap.cc:492] T 01402c21678c49ca9a143bb2272a549c P 3ee011df676c43f7beaeace0f8c60725: Bootstrap starting.
I20240501 07:59:35.656327 2154 tablet_bootstrap.cc:654] T 01402c21678c49ca9a143bb2272a549c P 3ee011df676c43f7beaeace0f8c60725: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.658841 2154 tablet_bootstrap.cc:492] T 01402c21678c49ca9a143bb2272a549c P 3ee011df676c43f7beaeace0f8c60725: No bootstrap required, opened a new log
I20240501 07:59:35.658931 2154 ts_tablet_manager.cc:1367] T 01402c21678c49ca9a143bb2272a549c P 3ee011df676c43f7beaeace0f8c60725: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:35.659314 2154 raft_consensus.cc:348] T 01402c21678c49ca9a143bb2272a549c P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.659416 2154 raft_consensus.cc:374] T 01402c21678c49ca9a143bb2272a549c P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.659451 2154 raft_consensus.cc:729] T 01402c21678c49ca9a143bb2272a549c P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee011df676c43f7beaeace0f8c60725, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.659600 2154 consensus_queue.cc:260] T 01402c21678c49ca9a143bb2272a549c P 3ee011df676c43f7beaeace0f8c60725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.659833 2154 ts_tablet_manager.cc:1398] T 01402c21678c49ca9a143bb2272a549c P 3ee011df676c43f7beaeace0f8c60725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:35.660506 2153 tablet_bootstrap.cc:492] T 15d5127ae40944f0a7ac301047245472 P f5af98ef61b54ae0b33faeb78af96dec: No bootstrap required, opened a new log
I20240501 07:59:35.660604 2153 ts_tablet_manager.cc:1367] T 15d5127ae40944f0a7ac301047245472 P f5af98ef61b54ae0b33faeb78af96dec: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 07:59:35.661007 2153 raft_consensus.cc:348] T 15d5127ae40944f0a7ac301047245472 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.661089 2153 raft_consensus.cc:374] T 15d5127ae40944f0a7ac301047245472 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.661110 2153 raft_consensus.cc:729] T 15d5127ae40944f0a7ac301047245472 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5af98ef61b54ae0b33faeb78af96dec, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.661213 2153 consensus_queue.cc:260] T 15d5127ae40944f0a7ac301047245472 P f5af98ef61b54ae0b33faeb78af96dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.661497 2153 ts_tablet_manager.cc:1398] T 15d5127ae40944f0a7ac301047245472 P f5af98ef61b54ae0b33faeb78af96dec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:35.662664 2151 tablet_bootstrap.cc:492] T 01402c21678c49ca9a143bb2272a549c P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:35.662768 2151 ts_tablet_manager.cc:1367] T 01402c21678c49ca9a143bb2272a549c P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 07:59:35.663246 2151 raft_consensus.cc:348] T 01402c21678c49ca9a143bb2272a549c P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.663373 2151 raft_consensus.cc:374] T 01402c21678c49ca9a143bb2272a549c P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.663396 2157 tablet_bootstrap.cc:492] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429: No bootstrap required, opened a new log
I20240501 07:59:35.663409 2151 raft_consensus.cc:729] T 01402c21678c49ca9a143bb2272a549c P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.663494 2157 ts_tablet_manager.cc:1367] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 07:59:35.663542 2151 consensus_queue.cc:260] T 01402c21678c49ca9a143bb2272a549c P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.663821 2151 ts_tablet_manager.cc:1398] T 01402c21678c49ca9a143bb2272a549c P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:35.663993 2151 tablet_bootstrap.cc:492] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c: Bootstrap starting.
I20240501 07:59:35.664580 2157 raft_consensus.cc:348] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.664687 2157 raft_consensus.cc:374] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.664718 2157 raft_consensus.cc:729] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45c306768564fb2bd741c535d334429, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.664959 2151 tablet_bootstrap.cc:654] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.664916 2157 consensus_queue.cc:260] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.665234 2157 ts_tablet_manager.cc:1398] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:35.666610 2151 tablet_bootstrap.cc:492] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c: No bootstrap required, opened a new log
I20240501 07:59:35.666708 2151 ts_tablet_manager.cc:1367] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:35.667199 2151 raft_consensus.cc:348] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:35.667296 2151 raft_consensus.cc:374] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.667327 2151 raft_consensus.cc:729] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.667439 2151 consensus_queue.cc:260] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:35.667703 2151 ts_tablet_manager.cc:1398] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:35.696274 2085 raft_consensus.cc:482] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:35.696416 2085 raft_consensus.cc:504] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:35.696949 2085 leader_election.cc:288] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:35.697412 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596a573d62964e01b0726168616f7f08" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429" is_pre_election: true
I20240501 07:59:35.697415 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596a573d62964e01b0726168616f7f08" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725" is_pre_election: true
I20240501 07:59:35.697546 1243 raft_consensus.cc:2454] T 596a573d62964e01b0726168616f7f08 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:35.697578 714 raft_consensus.cc:2454] T 596a573d62964e01b0726168616f7f08 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 0.
I20240501 07:59:35.697844 912 leader_election.cc:302] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:35.698042 2085 raft_consensus.cc:2781] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:35.698112 2085 raft_consensus.cc:482] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:35.698158 2085 raft_consensus.cc:3037] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.699272 2085 raft_consensus.cc:504] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:35.699617 2085 leader_election.cc:288] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Requested vote from peers 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765), c45c306768564fb2bd741c535d334429 (127.0.99.134:37067)
I20240501 07:59:35.699983 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596a573d62964e01b0726168616f7f08" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45c306768564fb2bd741c535d334429"
I20240501 07:59:35.700014 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596a573d62964e01b0726168616f7f08" candidate_uuid: "1526c5008f714bcfa69e9bacb51a662c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725"
I20240501 07:59:35.700105 1243 raft_consensus.cc:3037] T 596a573d62964e01b0726168616f7f08 P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.700132 714 raft_consensus.cc:3037] T 596a573d62964e01b0726168616f7f08 P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.700923 1243 raft_consensus.cc:2454] T 596a573d62964e01b0726168616f7f08 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:35.700945 714 raft_consensus.cc:2454] T 596a573d62964e01b0726168616f7f08 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1526c5008f714bcfa69e9bacb51a662c in term 1.
I20240501 07:59:35.701242 912 leader_election.cc:302] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:35.701421 2085 raft_consensus.cc:2781] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:35.701508 2085 raft_consensus.cc:686] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [term 1 LEADER]: Becoming Leader. State: Replica: 1526c5008f714bcfa69e9bacb51a662c, State: Running, Role: LEADER
I20240501 07:59:35.701660 2085 consensus_queue.cc:237] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } }
I20240501 07:59:35.703642 438 catalog_manager.cc:5484] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1526c5008f714bcfa69e9bacb51a662c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:35.704196 2198 raft_consensus.cc:482] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:35.704291 2198 raft_consensus.cc:504] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.704640 2198 leader_election.cc:288] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663), 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765)
I20240501 07:59:35.705039 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01402c21678c49ca9a143bb2272a549c" candidate_uuid: "c45c306768564fb2bd741c535d334429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725" is_pre_election: true
I20240501 07:59:35.705081 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01402c21678c49ca9a143bb2272a549c" candidate_uuid: "c45c306768564fb2bd741c535d334429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1526c5008f714bcfa69e9bacb51a662c" is_pre_election: true
I20240501 07:59:35.705180 714 raft_consensus.cc:2454] T 01402c21678c49ca9a143bb2272a549c P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c45c306768564fb2bd741c535d334429 in term 0.
I20240501 07:59:35.705197 979 raft_consensus.cc:2454] T 01402c21678c49ca9a143bb2272a549c P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c45c306768564fb2bd741c535d334429 in term 0.
I20240501 07:59:35.705500 1177 leader_election.cc:302] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee011df676c43f7beaeace0f8c60725, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:35.705737 2198 raft_consensus.cc:2781] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:35.705796 2198 raft_consensus.cc:482] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:35.705844 2198 raft_consensus.cc:3037] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.706653 2198 raft_consensus.cc:504] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.706954 2198 leader_election.cc:288] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [CANDIDATE]: Term 1 election: Requested vote from peers 1526c5008f714bcfa69e9bacb51a662c (127.0.99.132:36663), 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765)
I20240501 07:59:35.707332 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01402c21678c49ca9a143bb2272a549c" candidate_uuid: "c45c306768564fb2bd741c535d334429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1526c5008f714bcfa69e9bacb51a662c"
I20240501 07:59:35.707346 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01402c21678c49ca9a143bb2272a549c" candidate_uuid: "c45c306768564fb2bd741c535d334429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee011df676c43f7beaeace0f8c60725"
I20240501 07:59:35.707445 979 raft_consensus.cc:3037] T 01402c21678c49ca9a143bb2272a549c P 1526c5008f714bcfa69e9bacb51a662c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.707458 714 raft_consensus.cc:3037] T 01402c21678c49ca9a143bb2272a549c P 3ee011df676c43f7beaeace0f8c60725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.708235 714 raft_consensus.cc:2454] T 01402c21678c49ca9a143bb2272a549c P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c45c306768564fb2bd741c535d334429 in term 1.
I20240501 07:59:35.708252 979 raft_consensus.cc:2454] T 01402c21678c49ca9a143bb2272a549c P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c45c306768564fb2bd741c535d334429 in term 1.
I20240501 07:59:35.708523 1177 leader_election.cc:302] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1526c5008f714bcfa69e9bacb51a662c, c45c306768564fb2bd741c535d334429; no voters:
I20240501 07:59:35.708789 2198 raft_consensus.cc:2781] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:35.708889 2198 raft_consensus.cc:686] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [term 1 LEADER]: Becoming Leader. State: Replica: c45c306768564fb2bd741c535d334429, State: Running, Role: LEADER
I20240501 07:59:35.709028 2198 consensus_queue.cc:237] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } }
I20240501 07:59:35.710984 438 catalog_manager.cc:5484] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 reported cstate change: term changed from 0 to 1, leader changed from <none> to c45c306768564fb2bd741c535d334429 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "c45c306768564fb2bd741c535d334429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:35.809788 1740 raft_consensus.cc:482] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:35.809939 1740 raft_consensus.cc:504] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.810482 1740 leader_election.cc:288] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665), b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609)
I20240501 07:59:35.810914 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15d5127ae40944f0a7ac301047245472" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec" is_pre_election: true
I20240501 07:59:35.810952 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15d5127ae40944f0a7ac301047245472" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747" is_pre_election: true
I20240501 07:59:35.811072 1109 raft_consensus.cc:2454] T 15d5127ae40944f0a7ac301047245472 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:35.811072 582 raft_consensus.cc:2454] T 15d5127ae40944f0a7ac301047245472 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 0.
I20240501 07:59:35.811391 781 leader_election.cc:302] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, b9644aeae9de4deba0c3a2b85df04747; no voters:
I20240501 07:59:35.811571 1740 raft_consensus.cc:2781] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:35.811635 1740 raft_consensus.cc:482] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:35.811661 1740 raft_consensus.cc:3037] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.812543 1740 raft_consensus.cc:504] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.812857 1740 leader_election.cc:288] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Requested vote from peers f5af98ef61b54ae0b33faeb78af96dec (127.0.99.133:38665), b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609)
I20240501 07:59:35.813184 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15d5127ae40944f0a7ac301047245472" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9644aeae9de4deba0c3a2b85df04747"
I20240501 07:59:35.813314 582 raft_consensus.cc:3037] T 15d5127ae40944f0a7ac301047245472 P b9644aeae9de4deba0c3a2b85df04747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.813314 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15d5127ae40944f0a7ac301047245472" candidate_uuid: "8ee66e185465431389043d933cb51308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5af98ef61b54ae0b33faeb78af96dec"
I20240501 07:59:35.813418 1109 raft_consensus.cc:3037] T 15d5127ae40944f0a7ac301047245472 P f5af98ef61b54ae0b33faeb78af96dec [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.814347 582 raft_consensus.cc:2454] T 15d5127ae40944f0a7ac301047245472 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:35.814347 1109 raft_consensus.cc:2454] T 15d5127ae40944f0a7ac301047245472 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee66e185465431389043d933cb51308 in term 1.
I20240501 07:59:35.814674 781 leader_election.cc:302] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee66e185465431389043d933cb51308, b9644aeae9de4deba0c3a2b85df04747; no voters:
I20240501 07:59:35.814894 1740 raft_consensus.cc:2781] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:35.814996 1740 raft_consensus.cc:686] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee66e185465431389043d933cb51308, State: Running, Role: LEADER
I20240501 07:59:35.815168 1740 consensus_queue.cc:237] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } }
I20240501 07:59:35.817075 438 catalog_manager.cc:5484] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee66e185465431389043d933cb51308 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8ee66e185465431389043d933cb51308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee66e185465431389043d933cb51308" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37541 } health_report { overall_health: HEALTHY } } }
W20240501 07:59:35.968631 2216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:35.968936 2216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:35.972167 2216 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240501 07:59:36.057164 1243 raft_consensus.cc:1261] T 596a573d62964e01b0726168616f7f08 P c45c306768564fb2bd741c535d334429 [term 1 FOLLOWER]: Refusing update from remote peer 1526c5008f714bcfa69e9bacb51a662c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:36.057709 2085 consensus_queue.cc:1035] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45c306768564fb2bd741c535d334429" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:36.060851 714 raft_consensus.cc:1261] T 01402c21678c49ca9a143bb2272a549c P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Refusing update from remote peer c45c306768564fb2bd741c535d334429: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:36.061208 979 raft_consensus.cc:1261] T 01402c21678c49ca9a143bb2272a549c P 1526c5008f714bcfa69e9bacb51a662c [term 1 FOLLOWER]: Refusing update from remote peer c45c306768564fb2bd741c535d334429: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:36.062008 1109 raft_consensus.cc:1261] T 15d5127ae40944f0a7ac301047245472 P f5af98ef61b54ae0b33faeb78af96dec [term 1 FOLLOWER]: Refusing update from remote peer 8ee66e185465431389043d933cb51308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:36.063534 582 raft_consensus.cc:1261] T 15d5127ae40944f0a7ac301047245472 P b9644aeae9de4deba0c3a2b85df04747 [term 1 FOLLOWER]: Refusing update from remote peer 8ee66e185465431389043d933cb51308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:36.063894 2198 consensus_queue.cc:1035] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:36.063935 1740 consensus_queue.cc:1035] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5af98ef61b54ae0b33faeb78af96dec" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:36.064297 2198 consensus_queue.cc:1035] T 01402c21678c49ca9a143bb2272a549c P c45c306768564fb2bd741c535d334429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1526c5008f714bcfa69e9bacb51a662c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:36.064715 1740 consensus_queue.cc:1035] T 15d5127ae40944f0a7ac301047245472 P 8ee66e185465431389043d933cb51308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9644aeae9de4deba0c3a2b85df04747" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:36.070459 715 raft_consensus.cc:1261] T 596a573d62964e01b0726168616f7f08 P 3ee011df676c43f7beaeace0f8c60725 [term 1 FOLLOWER]: Refusing update from remote peer 1526c5008f714bcfa69e9bacb51a662c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:36.071087 2085 consensus_queue.cc:1035] T 596a573d62964e01b0726168616f7f08 P 1526c5008f714bcfa69e9bacb51a662c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee011df676c43f7beaeace0f8c60725" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:36.115454 2255 mvcc.cc:205] Tried to move back new op lower bound from 7022798340357210112 to 7022798338905432064. Current Snapshot: MvccSnapshot[applied={T|T < 7022798340337299456}]
INSERT report
rows total: 2000
time total: 108.939 ms
time per row: 0.0544696 ms
I20240501 07:59:36.349560 1222 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:36.349607 958 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:36.356678 695 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:36.359020 827 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:36.359901 563 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:36.360529 1091 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:36.497263 958 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:36.498651 827 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:36.501910 695 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:36.502727 1222 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:36.512586 1091 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:36.524441 563 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:36.657616 1222 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:36.661264 958 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:36.663020 563 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:36.671845 827 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:36.673522 1091 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:36.677209 695 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:36.793704 958 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:36.794144 695 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:36.794476 563 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:36.796428 827 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:36.805266 1222 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:36.808781 1091 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:36.951297 1222 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:36.951781 958 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:36.955576 563 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:36.956981 827 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:36.958191 695 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:36.973436 1091 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:37.118583 563 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:37.118583 958 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:37.118877 695 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:37.122483 1222 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:37.124152 1091 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:37.126806 827 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
1b115d81237d477ebb7ce82f63d5eac2 | 127.0.99.190:46217 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:34447 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
1526c5008f714bcfa69e9bacb51a662c | 127.0.99.132:36663 | HEALTHY | /B | 10 | 0
3ee011df676c43f7beaeace0f8c60725 | 127.0.99.130:40765 | HEALTHY | /A | 4 | 0
8ee66e185465431389043d933cb51308 | 127.0.99.131:37541 | HEALTHY | /B | 9 | 0
b9644aeae9de4deba0c3a2b85df04747 | 127.0.99.129:34609 | HEALTHY | /A | 5 | 0
c45c306768564fb2bd741c535d334429 | 127.0.99.134:37067 | HEALTHY | /C | 4 | 0
f5af98ef61b54ae0b33faeb78af96dec | 127.0.99.133:38665 | HEALTHY | /C | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/C | 2
/B | 2
/A | 2
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:34609
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:40765
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:37541
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:36663
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:38665
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:37067
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:34609
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:40765
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:37541
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:36663
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:38665
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:37067
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:34447 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_10 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_11 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 18
Median | 18
Third Quartile | 18
Maximum | 18
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 12
Tablets | 36
Replicas | 108
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 07:59:37.180747 398 log_verifier.cc:126] Checking tablet 01402c21678c49ca9a143bb2272a549c
I20240501 07:59:37.210814 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 01402c21678c49ca9a143bb2272a549c
I20240501 07:59:37.210911 398 log_verifier.cc:126] Checking tablet 0511b10a941e4dd084a0bc667cf5298d
I20240501 07:59:37.240391 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0511b10a941e4dd084a0bc667cf5298d
I20240501 07:59:37.240459 398 log_verifier.cc:126] Checking tablet 07a6844fd48f4ced93631486c4a29489
I20240501 07:59:37.268404 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 07a6844fd48f4ced93631486c4a29489
I20240501 07:59:37.268478 398 log_verifier.cc:126] Checking tablet 12eff9f6cf9545d1bbd326bcb0e84b22
I20240501 07:59:37.299265 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 12eff9f6cf9545d1bbd326bcb0e84b22
I20240501 07:59:37.299330 398 log_verifier.cc:126] Checking tablet 142a1237964a40d1b7e5ce852633be44
I20240501 07:59:37.326987 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 142a1237964a40d1b7e5ce852633be44
I20240501 07:59:37.327060 398 log_verifier.cc:126] Checking tablet 15d5127ae40944f0a7ac301047245472
I20240501 07:59:37.354933 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 15d5127ae40944f0a7ac301047245472
I20240501 07:59:37.355005 398 log_verifier.cc:126] Checking tablet 1e5764e3fc734f7e9380cb541d5b30e6
I20240501 07:59:37.383291 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1e5764e3fc734f7e9380cb541d5b30e6
I20240501 07:59:37.383368 398 log_verifier.cc:126] Checking tablet 1f0cf885cadd4d51960e4d198ed41679
I20240501 07:59:37.411499 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1f0cf885cadd4d51960e4d198ed41679
I20240501 07:59:37.411559 398 log_verifier.cc:126] Checking tablet 27db9a40cf144c4a9018e906ee72e54f
I20240501 07:59:37.440083 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 27db9a40cf144c4a9018e906ee72e54f
I20240501 07:59:37.440143 398 log_verifier.cc:126] Checking tablet 37732050f7a74053827b657e521f9a92
I20240501 07:59:37.467665 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 37732050f7a74053827b657e521f9a92
I20240501 07:59:37.467733 398 log_verifier.cc:126] Checking tablet 45765892e3df4768b034a748ec5e0980
I20240501 07:59:37.494951 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 45765892e3df4768b034a748ec5e0980
I20240501 07:59:37.495018 398 log_verifier.cc:126] Checking tablet 5240af8639074b7b92c38021d56b3482
I20240501 07:59:37.522552 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5240af8639074b7b92c38021d56b3482
I20240501 07:59:37.522624 398 log_verifier.cc:126] Checking tablet 549c8b9eef0440979653e95da1523966
I20240501 07:59:37.550190 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 549c8b9eef0440979653e95da1523966
I20240501 07:59:37.550261 398 log_verifier.cc:126] Checking tablet 57adc45a29584ef388c671fd4e7c4cf8
I20240501 07:59:37.577397 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 57adc45a29584ef388c671fd4e7c4cf8
I20240501 07:59:37.577463 398 log_verifier.cc:126] Checking tablet 5838f08a8fd04e2a9fcf192684503bf3
I20240501 07:59:37.605974 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5838f08a8fd04e2a9fcf192684503bf3
I20240501 07:59:37.606068 398 log_verifier.cc:126] Checking tablet 596a573d62964e01b0726168616f7f08
I20240501 07:59:37.637795 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 596a573d62964e01b0726168616f7f08
I20240501 07:59:37.637871 398 log_verifier.cc:126] Checking tablet 5a864679fa6948f5862417722e296302
I20240501 07:59:37.665201 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5a864679fa6948f5862417722e296302
I20240501 07:59:37.665268 398 log_verifier.cc:126] Checking tablet 699a8bf290e8422f97d4f828d03551eb
I20240501 07:59:37.693635 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 699a8bf290e8422f97d4f828d03551eb
I20240501 07:59:37.693715 398 log_verifier.cc:126] Checking tablet 7318525e5563450aa4875dc993637844
I20240501 07:59:37.722170 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7318525e5563450aa4875dc993637844
I20240501 07:59:37.722249 398 log_verifier.cc:126] Checking tablet 73ba1454ed0d4be0bf354a07c5fc7980
I20240501 07:59:37.750663 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 73ba1454ed0d4be0bf354a07c5fc7980
I20240501 07:59:37.750730 398 log_verifier.cc:126] Checking tablet 77f7440c254d4232b19565899d33f4a2
I20240501 07:59:37.779471 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 77f7440c254d4232b19565899d33f4a2
I20240501 07:59:37.779567 398 log_verifier.cc:126] Checking tablet 85220f32c33f4b7085852164e6b19add
I20240501 07:59:37.807505 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 85220f32c33f4b7085852164e6b19add
I20240501 07:59:37.807567 398 log_verifier.cc:126] Checking tablet 857f176f0500475894550b58d43e422b
I20240501 07:59:37.835752 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 857f176f0500475894550b58d43e422b
I20240501 07:59:37.835842 398 log_verifier.cc:126] Checking tablet 87d19c5333d448c0aa8440fff8ded2ef
I20240501 07:59:37.863890 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 87d19c5333d448c0aa8440fff8ded2ef
I20240501 07:59:37.863960 398 log_verifier.cc:126] Checking tablet 8d5c68ac4b29424e8f0dbb50dbdfb77f
I20240501 07:59:37.892592 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8d5c68ac4b29424e8f0dbb50dbdfb77f
I20240501 07:59:37.892659 398 log_verifier.cc:126] Checking tablet 8e85dcd82f4e462ba9d1e8965a5938e1
I20240501 07:59:37.921339 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8e85dcd82f4e462ba9d1e8965a5938e1
I20240501 07:59:37.921417 398 log_verifier.cc:126] Checking tablet 8f9333d0caf54332b6a6dc3fe643c35a
I20240501 07:59:37.949931 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8f9333d0caf54332b6a6dc3fe643c35a
I20240501 07:59:37.950024 398 log_verifier.cc:126] Checking tablet 9d2047f2e0094e99842cc3929571f39c
I20240501 07:59:37.977717 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9d2047f2e0094e99842cc3929571f39c
I20240501 07:59:37.977792 398 log_verifier.cc:126] Checking tablet a523579b0ba14640bfbd24d4fd2f5606
I20240501 07:59:38.005607 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a523579b0ba14640bfbd24d4fd2f5606
I20240501 07:59:38.005672 398 log_verifier.cc:126] Checking tablet ac6730a31bab4499a2c97f95ea217a09
I20240501 07:59:38.033406 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ac6730a31bab4499a2c97f95ea217a09
I20240501 07:59:38.033480 398 log_verifier.cc:126] Checking tablet c9138f65fa2a4d76ba9ee66119ade3ef
I20240501 07:59:38.062278 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c9138f65fa2a4d76ba9ee66119ade3ef
I20240501 07:59:38.062341 398 log_verifier.cc:126] Checking tablet c99a78b4f41244a9a95dd627ef356142
I20240501 07:59:38.090099 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c99a78b4f41244a9a95dd627ef356142
I20240501 07:59:38.090193 398 log_verifier.cc:126] Checking tablet d123616a55194df79645acca6e1e4ae8
I20240501 07:59:38.118491 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d123616a55194df79645acca6e1e4ae8
I20240501 07:59:38.118554 398 log_verifier.cc:126] Checking tablet dc96c465fe554a94aa599200d717a3c2
I20240501 07:59:38.146154 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet dc96c465fe554a94aa599200d717a3c2
I20240501 07:59:38.146219 398 log_verifier.cc:126] Checking tablet ead49854fd4d484db8b90d1e14ec3873
I20240501 07:59:38.174446 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ead49854fd4d484db8b90d1e14ec3873
I20240501 07:59:38.174516 398 log_verifier.cc:126] Checking tablet fbbab77fbfef40a08384f5d24526c8b4
I20240501 07:59:38.203238 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fbbab77fbfef40a08384f5d24526c8b4
I20240501 07:59:38.219429 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 501
W20240501 07:59:38.226656 781 connection.cc:535] client connection to 127.0.99.129:34609 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:38.226850 781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 07:59:38.227468 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 632
W20240501 07:59:38.228715 781 consensus_peers.cc:485] T d123616a55194df79645acca6e1e4ae8 P 8ee66e185465431389043d933cb51308 -> Peer b9644aeae9de4deba0c3a2b85df04747 (127.0.99.129:34609): Couldn't send request to peer b9644aeae9de4deba0c3a2b85df04747. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:34609: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:38.235051 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 764
W20240501 07:59:38.238597 913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.130:40765: connect: Connection refused (error 111)
W20240501 07:59:38.239493 913 consensus_peers.cc:485] T 07a6844fd48f4ced93631486c4a29489 P 1526c5008f714bcfa69e9bacb51a662c -> Peer 3ee011df676c43f7beaeace0f8c60725 (127.0.99.130:40765): Couldn't send request to peer 3ee011df676c43f7beaeace0f8c60725. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.130:40765: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:38.242362 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 896
I20240501 07:59:38.248751 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 1029
I20240501 07:59:38.255146 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 1160
I20240501 07:59:38.261787 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 410
2024-05-01T07:59:38Z chronyd exiting
[ OK ] LocationAwareRebalancingBasicTest.Basic (14977 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (14977 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240501 07:59:38.383109 398 test_util.cc:268] Using random seed: -566180098
I20240501 07:59:38.383922 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 07:59:38.383975 398 ts_itest-base.cc:113] --------------
I20240501 07:59:38.383986 398 ts_itest-base.cc:114] 5 tablet servers
I20240501 07:59:38.383997 398 ts_itest-base.cc:115] 3 replicas per TS
I20240501 07:59:38.384007 398 ts_itest-base.cc:116] --------------
2024-05-01T07:59:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T07:59:38Z Disabled control of system clock
I20240501 07:59:38.392359 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37959
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:46645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:37959
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 07:59:38.505110 2554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:38.505371 2554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:38.505438 2554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:38.508607 2554 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:38.508719 2554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 07:59:38.508755 2554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:38.508800 2554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 07:59:38.508846 2554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 07:59:38.512667 2554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:37959
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:38.513671 2554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:38.514969 2554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:38.523663 2559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:38.524619 2560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:38.525684 2562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:38.526582 2554 server_base.cc:1034] running on GCE node
I20240501 07:59:38.527182 2554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:38.528110 2554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:38.529258 2554 hybrid_clock.cc:648] HybridClock initialized: now 1714550378529246 us; error 29 us; skew 500 ppm
I20240501 07:59:38.531356 2554 webserver.cc:460] Webserver started at http://127.0.99.190:37441/ using document root <none> and password file <none>
I20240501 07:59:38.531929 2554 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:38.531986 2554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:38.532171 2554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:38.533872 2554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3d4ea92b7d2940018c2608bd80b7fa0f"
format_stamp: "Formatted at 2024-05-01 07:59:38 on dist-test-slave-wslg"
I20240501 07:59:38.534417 2554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3d4ea92b7d2940018c2608bd80b7fa0f"
format_stamp: "Formatted at 2024-05-01 07:59:38 on dist-test-slave-wslg"
I20240501 07:59:38.537752 2554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:38.539995 2568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:38.541056 2554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:38.541178 2554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3d4ea92b7d2940018c2608bd80b7fa0f"
format_stamp: "Formatted at 2024-05-01 07:59:38 on dist-test-slave-wslg"
I20240501 07:59:38.541268 2554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:38.553905 2554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:38.554605 2554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:38.554774 2554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:38.561522 2554 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:37959
I20240501 07:59:38.561564 2620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:37959 every 8 connection(s)
I20240501 07:59:38.562628 2554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 07:59:38.565280 2621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:38.567728 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 2554
I20240501 07:59:38.567845 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 07:59:38.571031 2621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f: Bootstrap starting.
I20240501 07:59:38.573617 2621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:38.574496 2621 log.cc:824] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:38.576459 2621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f: No bootstrap required, opened a new log
I20240501 07:59:38.579403 2621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4ea92b7d2940018c2608bd80b7fa0f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37959 } }
I20240501 07:59:38.579602 2621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:38.579638 2621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d4ea92b7d2940018c2608bd80b7fa0f, State: Initialized, Role: FOLLOWER
I20240501 07:59:38.580006 2621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4ea92b7d2940018c2608bd80b7fa0f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37959 } }
I20240501 07:59:38.580116 2621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:59:38.580159 2621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:59:38.580238 2621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:38.580978 2621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4ea92b7d2940018c2608bd80b7fa0f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37959 } }
I20240501 07:59:38.581292 2621 leader_election.cc:302] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d4ea92b7d2940018c2608bd80b7fa0f; no voters:
I20240501 07:59:38.581573 2621 leader_election.cc:288] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:59:38.581697 2626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:38.581912 2626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [term 1 LEADER]: Becoming Leader. State: Replica: 3d4ea92b7d2940018c2608bd80b7fa0f, State: Running, Role: LEADER
I20240501 07:59:38.582260 2626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4ea92b7d2940018c2608bd80b7fa0f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37959 } }
I20240501 07:59:38.582667 2621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:59:38.583688 2628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d4ea92b7d2940018c2608bd80b7fa0f. Latest consensus state: current_term: 1 leader_uuid: "3d4ea92b7d2940018c2608bd80b7fa0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4ea92b7d2940018c2608bd80b7fa0f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37959 } } }
I20240501 07:59:38.583818 2628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:38.584156 2627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d4ea92b7d2940018c2608bd80b7fa0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4ea92b7d2940018c2608bd80b7fa0f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37959 } } }
I20240501 07:59:38.584255 2627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:38.584508 2633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:59:38.587533 2633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:59:38.592576 2633 catalog_manager.cc:1347] Generated new cluster ID: b6707c176f1e4d03b6acac920356c7a4
I20240501 07:59:38.592649 2633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:59:38.609256 2633 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:59:38.610258 2633 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:59:38.622742 2633 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3d4ea92b7d2940018c2608bd80b7fa0f: Generated new TSK 0
I20240501 07:59:38.623692 2633 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:59:38.630999 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37959
--builtin_ntp_servers=127.0.99.148:46645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 07:59:38.741878 2645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:38.742165 2645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:38.742228 2645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:38.745426 2645 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:38.745496 2645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:38.745599 2645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:59:38.749289 2645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37959
--never_fsync=true
--heap_profile_path=/tmp/kudu.2645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:38.750247 2645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:38.751394 2645 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:38.760720 2651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:38.761443 2650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:38.762367 2653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:38.763636 2645 server_base.cc:1034] running on GCE node
I20240501 07:59:38.764112 2645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:38.764735 2645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:38.765897 2645 hybrid_clock.cc:648] HybridClock initialized: now 1714550378765876 us; error 38 us; skew 500 ppm
I20240501 07:59:38.768009 2645 webserver.cc:460] Webserver started at http://127.0.99.129:37845/ using document root <none> and password file <none>
I20240501 07:59:38.768606 2645 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:38.768672 2645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:38.768862 2645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:38.770710 2645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "bcdf278f9d314be6971bbd6533436716"
format_stamp: "Formatted at 2024-05-01 07:59:38 on dist-test-slave-wslg"
I20240501 07:59:38.771354 2645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "bcdf278f9d314be6971bbd6533436716"
format_stamp: "Formatted at 2024-05-01 07:59:38 on dist-test-slave-wslg"
I20240501 07:59:38.775089 2645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240501 07:59:38.777642 2659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:38.778851 2645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:38.778980 2645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "bcdf278f9d314be6971bbd6533436716"
format_stamp: "Formatted at 2024-05-01 07:59:38 on dist-test-slave-wslg"
I20240501 07:59:38.779093 2645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:38.796525 2645 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:38.797211 2645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:38.797389 2645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:38.797931 2645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:38.798851 2645 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:38.798902 2645 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:38.798964 2645 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:38.798991 2645 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:38.807438 2645 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:33429
I20240501 07:59:38.807477 2772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:33429 every 8 connection(s)
I20240501 07:59:38.808408 2645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:59:38.817258 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 2645
I20240501 07:59:38.817405 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 07:59:38.817485 2773 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37959
I20240501 07:59:38.817695 2773 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:38.818367 2773 heartbeater.cc:511] Master 127.0.99.190:37959 requested a full tablet report, sending...
I20240501 07:59:38.819737 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37959
--builtin_ntp_servers=127.0.99.148:46645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:38.820048 2585 ts_manager.cc:194] Registered new tserver with Master: bcdf278f9d314be6971bbd6533436716 (127.0.99.129:33429)
I20240501 07:59:38.821615 2585 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:42271
W20240501 07:59:38.931622 2776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:38.931949 2776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:38.932044 2776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:38.935303 2776 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:38.935400 2776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:38.935505 2776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 07:59:38.939301 2776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37959
--never_fsync=true
--heap_profile_path=/tmp/kudu.2776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:38.940325 2776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:38.941483 2776 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:38.950954 2781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:38.952062 2782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:38.952814 2784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:38.953619 2776 server_base.cc:1034] running on GCE node
I20240501 07:59:38.954123 2776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:38.954762 2776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:38.955969 2776 hybrid_clock.cc:648] HybridClock initialized: now 1714550378955959 us; error 35 us; skew 500 ppm
I20240501 07:59:38.958273 2776 webserver.cc:460] Webserver started at http://127.0.99.130:39245/ using document root <none> and password file <none>
I20240501 07:59:38.958846 2776 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:38.958907 2776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:38.959085 2776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:38.960785 2776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "da4aa47b096e46dda582839503d12d8e"
format_stamp: "Formatted at 2024-05-01 07:59:38 on dist-test-slave-wslg"
I20240501 07:59:38.961298 2776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "da4aa47b096e46dda582839503d12d8e"
format_stamp: "Formatted at 2024-05-01 07:59:38 on dist-test-slave-wslg"
I20240501 07:59:38.964933 2776 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:59:38.967504 2790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:38.968669 2776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:38.968813 2776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "da4aa47b096e46dda582839503d12d8e"
format_stamp: "Formatted at 2024-05-01 07:59:38 on dist-test-slave-wslg"
I20240501 07:59:38.968916 2776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:38.983971 2776 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:38.984704 2776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:38.984872 2776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:38.985448 2776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:38.986410 2776 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:38.986464 2776 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:38.986524 2776 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:38.986555 2776 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:38.995270 2776 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:44543
I20240501 07:59:38.995292 2903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:44543 every 8 connection(s)
I20240501 07:59:38.996292 2776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 07:59:39.005635 2904 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37959
I20240501 07:59:39.005887 2904 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:39.006253 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 2776
I20240501 07:59:39.006340 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 07:59:39.006527 2904 heartbeater.cc:511] Master 127.0.99.190:37959 requested a full tablet report, sending...
I20240501 07:59:39.007582 2584 ts_manager.cc:194] Registered new tserver with Master: da4aa47b096e46dda582839503d12d8e (127.0.99.130:44543)
I20240501 07:59:39.008308 2584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:56269
I20240501 07:59:39.008603 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37959
--builtin_ntp_servers=127.0.99.148:46645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 07:59:39.117195 2907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:39.117449 2907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:39.117527 2907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:39.120662 2907 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:39.120733 2907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:39.120805 2907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:59:39.124644 2907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37959
--never_fsync=true
--heap_profile_path=/tmp/kudu.2907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:39.125623 2907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:39.126799 2907 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:39.135993 2913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:39.137024 2912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:39.138180 2915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:39.139065 2907 server_base.cc:1034] running on GCE node
I20240501 07:59:39.139528 2907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:39.140154 2907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:39.141335 2907 hybrid_clock.cc:648] HybridClock initialized: now 1714550379141319 us; error 29 us; skew 500 ppm
I20240501 07:59:39.143455 2907 webserver.cc:460] Webserver started at http://127.0.99.131:42775/ using document root <none> and password file <none>
I20240501 07:59:39.144069 2907 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:39.144143 2907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:39.144345 2907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:39.146205 2907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "61f6dc94427e418b9e9708e223135259"
format_stamp: "Formatted at 2024-05-01 07:59:39 on dist-test-slave-wslg"
I20240501 07:59:39.146684 2907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "61f6dc94427e418b9e9708e223135259"
format_stamp: "Formatted at 2024-05-01 07:59:39 on dist-test-slave-wslg"
I20240501 07:59:39.150242 2907 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:59:39.152462 2921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:39.153475 2907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:39.153575 2907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "61f6dc94427e418b9e9708e223135259"
format_stamp: "Formatted at 2024-05-01 07:59:39 on dist-test-slave-wslg"
I20240501 07:59:39.153669 2907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:39.166580 2907 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:39.167261 2907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:39.167423 2907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:39.167943 2907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:39.168825 2907 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:39.168874 2907 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:39.168913 2907 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:39.168946 2907 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:39.177709 2907 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:46291
I20240501 07:59:39.177754 3034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:46291 every 8 connection(s)
I20240501 07:59:39.178705 2907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:59:39.184629 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 2907
I20240501 07:59:39.184713 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 07:59:39.186944 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37959
--builtin_ntp_servers=127.0.99.148:46645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:39.188185 3035 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37959
I20240501 07:59:39.188443 3035 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:39.189143 3035 heartbeater.cc:511] Master 127.0.99.190:37959 requested a full tablet report, sending...
I20240501 07:59:39.190270 2584 ts_manager.cc:194] Registered new tserver with Master: 61f6dc94427e418b9e9708e223135259 (127.0.99.131:46291)
I20240501 07:59:39.190914 2584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:46961
W20240501 07:59:39.297348 3038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:39.297618 3038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:39.297703 3038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:39.300894 3038 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:39.300987 3038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:39.301096 3038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:59:39.304931 3038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37959
--never_fsync=true
--heap_profile_path=/tmp/kudu.3038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:39.305966 3038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:39.307284 3038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:39.316509 3046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:39.317230 3043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:39.318351 3044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:39.319355 3038 server_base.cc:1034] running on GCE node
I20240501 07:59:39.319871 3038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:39.320542 3038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:39.321702 3038 hybrid_clock.cc:648] HybridClock initialized: now 1714550379321690 us; error 29 us; skew 500 ppm
I20240501 07:59:39.323942 3038 webserver.cc:460] Webserver started at http://127.0.99.132:33679/ using document root <none> and password file <none>
I20240501 07:59:39.324518 3038 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:39.324580 3038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:39.324761 3038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:39.326547 3038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "7fc9f166b03d4bcd804d76fd882b693c"
format_stamp: "Formatted at 2024-05-01 07:59:39 on dist-test-slave-wslg"
I20240501 07:59:39.327026 3038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "7fc9f166b03d4bcd804d76fd882b693c"
format_stamp: "Formatted at 2024-05-01 07:59:39 on dist-test-slave-wslg"
I20240501 07:59:39.330430 3038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:59:39.332710 3052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:39.333760 3038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:39.333911 3038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7fc9f166b03d4bcd804d76fd882b693c"
format_stamp: "Formatted at 2024-05-01 07:59:39 on dist-test-slave-wslg"
I20240501 07:59:39.334040 3038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:39.362851 3038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:39.363495 3038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:39.363639 3038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:39.364128 3038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:39.364959 3038 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:39.365005 3038 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:39.365060 3038 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:39.365089 3038 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:39.373697 3038 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:34433
I20240501 07:59:39.373746 3165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:34433 every 8 connection(s)
I20240501 07:59:39.374742 3038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:59:39.383832 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 3038
I20240501 07:59:39.383957 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 07:59:39.384130 3166 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37959
I20240501 07:59:39.384491 3166 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:39.385185 3166 heartbeater.cc:511] Master 127.0.99.190:37959 requested a full tablet report, sending...
I20240501 07:59:39.386155 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37959
--builtin_ntp_servers=127.0.99.148:46645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:39.386224 2584 ts_manager.cc:194] Registered new tserver with Master: 7fc9f166b03d4bcd804d76fd882b693c (127.0.99.132:34433)
I20240501 07:59:39.386972 2584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:57117
W20240501 07:59:39.497119 3169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:39.497372 3169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:39.497418 3169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:39.500537 3169 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:39.500624 3169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:39.500696 3169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:59:39.504505 3169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37959
--never_fsync=true
--heap_profile_path=/tmp/kudu.3169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:39.505534 3169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:39.506716 3169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:39.515749 3177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:39.516556 3174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:39.517663 3175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:39.518700 3169 server_base.cc:1034] running on GCE node
I20240501 07:59:39.519109 3169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:39.519681 3169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:39.520874 3169 hybrid_clock.cc:648] HybridClock initialized: now 1714550379520848 us; error 49 us; skew 500 ppm
I20240501 07:59:39.522941 3169 webserver.cc:460] Webserver started at http://127.0.99.133:36801/ using document root <none> and password file <none>
I20240501 07:59:39.523487 3169 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:39.523540 3169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:39.523692 3169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:39.525374 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "5507ab2319784caaaeb769e4518da1b9"
format_stamp: "Formatted at 2024-05-01 07:59:39 on dist-test-slave-wslg"
I20240501 07:59:39.525815 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "5507ab2319784caaaeb769e4518da1b9"
format_stamp: "Formatted at 2024-05-01 07:59:39 on dist-test-slave-wslg"
I20240501 07:59:39.529259 3169 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240501 07:59:39.531512 3183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:39.532606 3169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:39.532735 3169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5507ab2319784caaaeb769e4518da1b9"
format_stamp: "Formatted at 2024-05-01 07:59:39 on dist-test-slave-wslg"
I20240501 07:59:39.532842 3169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:39.546512 3169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:39.547154 3169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:39.547320 3169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:39.547853 3169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:39.548740 3169 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:39.548794 3169 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:39.548861 3169 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:39.548890 3169 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:39.557662 3169 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:38661
I20240501 07:59:39.557704 3296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:38661 every 8 connection(s)
I20240501 07:59:39.558748 3169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:59:39.562819 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 3169
I20240501 07:59:39.562944 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 07:59:39.569216 3297 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37959
I20240501 07:59:39.569489 3297 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:39.570202 3297 heartbeater.cc:511] Master 127.0.99.190:37959 requested a full tablet report, sending...
I20240501 07:59:39.571246 2584 ts_manager.cc:194] Registered new tserver with Master: 5507ab2319784caaaeb769e4518da1b9 (127.0.99.133:38661)
I20240501 07:59:39.571791 2584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:50111
I20240501 07:59:39.576855 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 07:59:39.586117 2584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60044:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 07:59:39.604213 3231 tablet_service.cc:1467] Processing CreateTablet for tablet 8b90f34d90c04b56a03a522dc5a309fa (DEFAULT_TABLE table=TestTable [id=772a1a2176d24c04ada32649413ec0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:39.604300 2836 tablet_service.cc:1467] Processing CreateTablet for tablet 8b90f34d90c04b56a03a522dc5a309fa (DEFAULT_TABLE table=TestTable [id=772a1a2176d24c04ada32649413ec0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:39.604314 2967 tablet_service.cc:1467] Processing CreateTablet for tablet 8b90f34d90c04b56a03a522dc5a309fa (DEFAULT_TABLE table=TestTable [id=772a1a2176d24c04ada32649413ec0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:39.605211 2967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b90f34d90c04b56a03a522dc5a309fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:39.605211 2836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b90f34d90c04b56a03a522dc5a309fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:39.605209 3231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b90f34d90c04b56a03a522dc5a309fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:39.610535 3314 tablet_bootstrap.cc:492] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9: Bootstrap starting.
I20240501 07:59:39.611577 3315 tablet_bootstrap.cc:492] T 8b90f34d90c04b56a03a522dc5a309fa P da4aa47b096e46dda582839503d12d8e: Bootstrap starting.
I20240501 07:59:39.611723 3316 tablet_bootstrap.cc:492] T 8b90f34d90c04b56a03a522dc5a309fa P 61f6dc94427e418b9e9708e223135259: Bootstrap starting.
I20240501 07:59:39.612612 3314 tablet_bootstrap.cc:654] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:39.613287 3314 log.cc:824] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:39.613626 3316 tablet_bootstrap.cc:654] T 8b90f34d90c04b56a03a522dc5a309fa P 61f6dc94427e418b9e9708e223135259: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:39.613626 3315 tablet_bootstrap.cc:654] T 8b90f34d90c04b56a03a522dc5a309fa P da4aa47b096e46dda582839503d12d8e: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:39.614470 3315 log.cc:824] T 8b90f34d90c04b56a03a522dc5a309fa P da4aa47b096e46dda582839503d12d8e: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:39.614470 3316 log.cc:824] T 8b90f34d90c04b56a03a522dc5a309fa P 61f6dc94427e418b9e9708e223135259: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:39.615065 3314 tablet_bootstrap.cc:492] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9: No bootstrap required, opened a new log
I20240501 07:59:39.615240 3314 ts_tablet_manager.cc:1367] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 07:59:39.615960 3315 tablet_bootstrap.cc:492] T 8b90f34d90c04b56a03a522dc5a309fa P da4aa47b096e46dda582839503d12d8e: No bootstrap required, opened a new log
I20240501 07:59:39.616147 3315 ts_tablet_manager.cc:1367] T 8b90f34d90c04b56a03a522dc5a309fa P da4aa47b096e46dda582839503d12d8e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 07:59:39.618162 3314 raft_consensus.cc:348] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5507ab2319784caaaeb769e4518da1b9" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38661 } } peers { permanent_uuid: "61f6dc94427e418b9e9708e223135259" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46291 } } peers { permanent_uuid: "da4aa47b096e46dda582839503d12d8e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44543 } }
I20240501 07:59:39.618350 3314 raft_consensus.cc:374] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:39.618433 3314 raft_consensus.cc:729] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5507ab2319784caaaeb769e4518da1b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:39.618880 3314 consensus_queue.cc:260] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5507ab2319784caaaeb769e4518da1b9" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38661 } } peers { permanent_uuid: "61f6dc94427e418b9e9708e223135259" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46291 } } peers { permanent_uuid: "da4aa47b096e46dda582839503d12d8e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44543 } }
I20240501 07:59:39.619268 3316 tablet_bootstrap.cc:492] T 8b90f34d90c04b56a03a522dc5a309fa P 61f6dc94427e418b9e9708e223135259: No bootstrap required, opened a new log
I20240501 07:59:39.619158 3315 raft_consensus.cc:348] T 8b90f34d90c04b56a03a522dc5a309fa P da4aa47b096e46dda582839503d12d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5507ab2319784caaaeb769e4518da1b9" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38661 } } peers { permanent_uuid: "61f6dc94427e418b9e9708e223135259" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46291 } } peers { permanent_uuid: "da4aa47b096e46dda582839503d12d8e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44543 } }
I20240501 07:59:39.619423 3315 raft_consensus.cc:374] T 8b90f34d90c04b56a03a522dc5a309fa P da4aa47b096e46dda582839503d12d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:39.619506 3316 ts_tablet_manager.cc:1367] T 8b90f34d90c04b56a03a522dc5a309fa P 61f6dc94427e418b9e9708e223135259: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240501 07:59:39.619518 3315 raft_consensus.cc:729] T 8b90f34d90c04b56a03a522dc5a309fa P da4aa47b096e46dda582839503d12d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da4aa47b096e46dda582839503d12d8e, State: Initialized, Role: FOLLOWER
I20240501 07:59:39.619726 3314 ts_tablet_manager.cc:1398] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 07:59:39.620097 3315 consensus_queue.cc:260] T 8b90f34d90c04b56a03a522dc5a309fa P da4aa47b096e46dda582839503d12d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5507ab2319784caaaeb769e4518da1b9" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38661 } } peers { permanent_uuid: "61f6dc94427e418b9e9708e223135259" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46291 } } peers { permanent_uuid: "da4aa47b096e46dda582839503d12d8e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44543 } }
I20240501 07:59:39.620337 3297 heartbeater.cc:503] Master 127.0.99.190:37959 was elected leader, sending a full tablet report...
I20240501 07:59:39.620810 2904 heartbeater.cc:503] Master 127.0.99.190:37959 was elected leader, sending a full tablet report...
I20240501 07:59:39.620993 3315 ts_tablet_manager.cc:1398] T 8b90f34d90c04b56a03a522dc5a309fa P da4aa47b096e46dda582839503d12d8e: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240501 07:59:39.623234 3316 raft_consensus.cc:348] T 8b90f34d90c04b56a03a522dc5a309fa P 61f6dc94427e418b9e9708e223135259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5507ab2319784caaaeb769e4518da1b9" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38661 } } peers { permanent_uuid: "61f6dc94427e418b9e9708e223135259" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46291 } } peers { permanent_uuid: "da4aa47b096e46dda582839503d12d8e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44543 } }
I20240501 07:59:39.623464 3316 raft_consensus.cc:374] T 8b90f34d90c04b56a03a522dc5a309fa P 61f6dc94427e418b9e9708e223135259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:39.623605 3316 raft_consensus.cc:729] T 8b90f34d90c04b56a03a522dc5a309fa P 61f6dc94427e418b9e9708e223135259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f6dc94427e418b9e9708e223135259, State: Initialized, Role: FOLLOWER
I20240501 07:59:39.624086 3316 consensus_queue.cc:260] T 8b90f34d90c04b56a03a522dc5a309fa P 61f6dc94427e418b9e9708e223135259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5507ab2319784caaaeb769e4518da1b9" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38661 } } peers { permanent_uuid: "61f6dc94427e418b9e9708e223135259" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46291 } } peers { permanent_uuid: "da4aa47b096e46dda582839503d12d8e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44543 } }
I20240501 07:59:39.624789 3035 heartbeater.cc:503] Master 127.0.99.190:37959 was elected leader, sending a full tablet report...
I20240501 07:59:39.625106 3316 ts_tablet_manager.cc:1398] T 8b90f34d90c04b56a03a522dc5a309fa P 61f6dc94427e418b9e9708e223135259: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:59:39.626097 3320 raft_consensus.cc:482] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:39.626253 3320 raft_consensus.cc:504] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5507ab2319784caaaeb769e4518da1b9" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38661 } } peers { permanent_uuid: "61f6dc94427e418b9e9708e223135259" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46291 } } peers { permanent_uuid: "da4aa47b096e46dda582839503d12d8e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44543 } }
I20240501 07:59:39.627348 3320 leader_election.cc:288] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61f6dc94427e418b9e9708e223135259 (127.0.99.131:46291), da4aa47b096e46dda582839503d12d8e (127.0.99.130:44543)
I20240501 07:59:39.631417 2858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b90f34d90c04b56a03a522dc5a309fa" candidate_uuid: "5507ab2319784caaaeb769e4518da1b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da4aa47b096e46dda582839503d12d8e" is_pre_election: true
I20240501 07:59:39.631732 2858 raft_consensus.cc:2454] T 8b90f34d90c04b56a03a522dc5a309fa P da4aa47b096e46dda582839503d12d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5507ab2319784caaaeb769e4518da1b9 in term 0.
I20240501 07:59:39.632054 2989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b90f34d90c04b56a03a522dc5a309fa" candidate_uuid: "5507ab2319784caaaeb769e4518da1b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f6dc94427e418b9e9708e223135259" is_pre_election: true
I20240501 07:59:39.632212 3187 leader_election.cc:302] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5507ab2319784caaaeb769e4518da1b9, da4aa47b096e46dda582839503d12d8e; no voters:
I20240501 07:59:39.632299 2989 raft_consensus.cc:2454] T 8b90f34d90c04b56a03a522dc5a309fa P 61f6dc94427e418b9e9708e223135259 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5507ab2319784caaaeb769e4518da1b9 in term 0.
I20240501 07:59:39.632459 3320 raft_consensus.cc:2781] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:39.632540 3320 raft_consensus.cc:482] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:39.632602 3320 raft_consensus.cc:3037] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:39.633566 3320 raft_consensus.cc:504] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5507ab2319784caaaeb769e4518da1b9" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38661 } } peers { permanent_uuid: "61f6dc94427e418b9e9708e223135259" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46291 } } peers { permanent_uuid: "da4aa47b096e46dda582839503d12d8e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44543 } }
I20240501 07:59:39.633900 3320 leader_election.cc:288] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [CANDIDATE]: Term 1 election: Requested vote from peers 61f6dc94427e418b9e9708e223135259 (127.0.99.131:46291), da4aa47b096e46dda582839503d12d8e (127.0.99.130:44543)
I20240501 07:59:39.634222 2989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b90f34d90c04b56a03a522dc5a309fa" candidate_uuid: "5507ab2319784caaaeb769e4518da1b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f6dc94427e418b9e9708e223135259"
I20240501 07:59:39.634284 2858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b90f34d90c04b56a03a522dc5a309fa" candidate_uuid: "5507ab2319784caaaeb769e4518da1b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da4aa47b096e46dda582839503d12d8e"
I20240501 07:59:39.634321 2989 raft_consensus.cc:3037] T 8b90f34d90c04b56a03a522dc5a309fa P 61f6dc94427e418b9e9708e223135259 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:39.634409 2858 raft_consensus.cc:3037] T 8b90f34d90c04b56a03a522dc5a309fa P da4aa47b096e46dda582839503d12d8e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:39.635267 2989 raft_consensus.cc:2454] T 8b90f34d90c04b56a03a522dc5a309fa P 61f6dc94427e418b9e9708e223135259 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5507ab2319784caaaeb769e4518da1b9 in term 1.
I20240501 07:59:39.635489 2858 raft_consensus.cc:2454] T 8b90f34d90c04b56a03a522dc5a309fa P da4aa47b096e46dda582839503d12d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5507ab2319784caaaeb769e4518da1b9 in term 1.
I20240501 07:59:39.635551 3184 leader_election.cc:302] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5507ab2319784caaaeb769e4518da1b9, 61f6dc94427e418b9e9708e223135259; no voters:
I20240501 07:59:39.635732 3320 raft_consensus.cc:2781] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:39.635962 3320 raft_consensus.cc:686] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [term 1 LEADER]: Becoming Leader. State: Replica: 5507ab2319784caaaeb769e4518da1b9, State: Running, Role: LEADER
I20240501 07:59:39.636286 3320 consensus_queue.cc:237] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5507ab2319784caaaeb769e4518da1b9" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38661 } } peers { permanent_uuid: "61f6dc94427e418b9e9708e223135259" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46291 } } peers { permanent_uuid: "da4aa47b096e46dda582839503d12d8e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44543 } }
I20240501 07:59:39.639318 2585 catalog_manager.cc:5484] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5507ab2319784caaaeb769e4518da1b9 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "5507ab2319784caaaeb769e4518da1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5507ab2319784caaaeb769e4518da1b9" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61f6dc94427e418b9e9708e223135259" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da4aa47b096e46dda582839503d12d8e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44543 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:39.649034 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 07:59:39.650499 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver bcdf278f9d314be6971bbd6533436716 to finish bootstrapping
I20240501 07:59:39.656574 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver da4aa47b096e46dda582839503d12d8e to finish bootstrapping
I20240501 07:59:39.661967 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 61f6dc94427e418b9e9708e223135259 to finish bootstrapping
I20240501 07:59:39.667258 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7fc9f166b03d4bcd804d76fd882b693c to finish bootstrapping
I20240501 07:59:39.672998 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5507ab2319784caaaeb769e4518da1b9 to finish bootstrapping
W20240501 07:59:39.679796 3036 tablet.cc:2369] T 8b90f34d90c04b56a03a522dc5a309fa P 61f6dc94427e418b9e9708e223135259: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:59:39.747561 2905 tablet.cc:2369] T 8b90f34d90c04b56a03a522dc5a309fa P da4aa47b096e46dda582839503d12d8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:59:39.809631 3298 tablet.cc:2369] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:39.824149 2773 heartbeater.cc:503] Master 127.0.99.190:37959 was elected leader, sending a full tablet report...
I20240501 07:59:39.866060 2703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:39.866590 3231 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.867002 3100 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:39.868408 2967 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:39.870512 2836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:39.957490 2703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:39.958022 3231 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.959085 2967 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:39.960369 3100 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:39.961061 2836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:40.037835 3320 consensus_queue.cc:1035] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da4aa47b096e46dda582839503d12d8e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:40.043716 3398 consensus_queue.cc:1035] T 8b90f34d90c04b56a03a522dc5a309fa P 5507ab2319784caaaeb769e4518da1b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61f6dc94427e418b9e9708e223135259" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:40.191721 2967 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:40.191731 3231 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:40.196453 2836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:40.199172 2703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:40.199561 3100 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:40.290963 2967 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:40.291275 3100 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:40.292374 3231 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:40.295162 2836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:40.295586 2703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:40.389292 3166 heartbeater.cc:503] Master 127.0.99.190:37959 was elected leader, sending a full tablet report...
I20240501 07:59:40.392807 2967 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:40.393078 3231 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:40.395107 3100 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:40.395578 2703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:40.399915 2836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:40.438017 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 2645
I20240501 07:59:40.443660 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 2776
I20240501 07:59:40.449384 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 2907
I20240501 07:59:40.455034 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 3038
I20240501 07:59:40.460420 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 3169
I20240501 07:59:40.466370 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 2554
2024-05-01T07:59:40Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (2106 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2106 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/3
I20240501 07:59:40.489586 398 test_util.cc:268] Using random seed: -564073621
I20240501 07:59:40.490583 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 07:59:40.490643 398 ts_itest-base.cc:113] --------------
I20240501 07:59:40.490662 398 ts_itest-base.cc:114] 7 tablet servers
I20240501 07:59:40.490677 398 ts_itest-base.cc:115] 2 replicas per TS
I20240501 07:59:40.490692 398 ts_itest-base.cc:116] --------------
2024-05-01T07:59:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T07:59:40Z Disabled control of system clock
I20240501 07:59:40.499429 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:44911
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:44911
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 07:59:40.608088 3506 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 07:59:40.608350 3506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:40.608383 3506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:40.608450 3506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:40.611510 3506 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:40.611584 3506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 07:59:40.611608 3506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:40.611625 3506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 07:59:40.611660 3506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 07:59:40.615782 3506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:44911
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:44911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.3506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:40.616779 3506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:40.618000 3506 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:40.626432 3511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:40.627143 3514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:40.628408 3512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:40.629424 3506 server_base.cc:1034] running on GCE node
I20240501 07:59:40.630053 3506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:40.631047 3506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:40.632200 3506 hybrid_clock.cc:648] HybridClock initialized: now 1714550380632191 us; error 27 us; skew 500 ppm
I20240501 07:59:40.634261 3506 webserver.cc:460] Webserver started at http://127.0.99.190:40485/ using document root <none> and password file <none>
I20240501 07:59:40.634833 3506 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:40.634891 3506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:40.635085 3506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:40.636785 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2033c6e9bc084ce3baa7be66008fcca6"
format_stamp: "Formatted at 2024-05-01 07:59:40 on dist-test-slave-wslg"
I20240501 07:59:40.637274 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2033c6e9bc084ce3baa7be66008fcca6"
format_stamp: "Formatted at 2024-05-01 07:59:40 on dist-test-slave-wslg"
I20240501 07:59:40.640581 3506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20240501 07:59:40.642930 3520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:40.644006 3506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:40.644133 3506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2033c6e9bc084ce3baa7be66008fcca6"
format_stamp: "Formatted at 2024-05-01 07:59:40 on dist-test-slave-wslg"
I20240501 07:59:40.644232 3506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:40.679085 3506 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:40.679792 3506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:40.679980 3506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:40.686640 3572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:44911 every 8 connection(s)
I20240501 07:59:40.686645 3506 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:44911
I20240501 07:59:40.687695 3506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 07:59:40.690454 3573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:40.695583 3573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6: Bootstrap starting.
I20240501 07:59:40.695673 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 3506
I20240501 07:59:40.695803 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 07:59:40.698097 3573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:40.698954 3573 log.cc:824] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:40.700660 3573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6: No bootstrap required, opened a new log
I20240501 07:59:40.703500 3573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2033c6e9bc084ce3baa7be66008fcca6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44911 } }
I20240501 07:59:40.703722 3573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:40.703786 3573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2033c6e9bc084ce3baa7be66008fcca6, State: Initialized, Role: FOLLOWER
I20240501 07:59:40.704301 3573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2033c6e9bc084ce3baa7be66008fcca6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44911 } }
I20240501 07:59:40.704437 3573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:59:40.704488 3573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:59:40.704589 3573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:40.705396 3573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2033c6e9bc084ce3baa7be66008fcca6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44911 } }
I20240501 07:59:40.705754 3573 leader_election.cc:302] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2033c6e9bc084ce3baa7be66008fcca6; no voters:
I20240501 07:59:40.706105 3573 leader_election.cc:288] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:59:40.706496 3578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:40.706693 3578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [term 1 LEADER]: Becoming Leader. State: Replica: 2033c6e9bc084ce3baa7be66008fcca6, State: Running, Role: LEADER
I20240501 07:59:40.706965 3578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2033c6e9bc084ce3baa7be66008fcca6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44911 } }
I20240501 07:59:40.707224 3573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:59:40.708456 3579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2033c6e9bc084ce3baa7be66008fcca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2033c6e9bc084ce3baa7be66008fcca6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44911 } } }
I20240501 07:59:40.708559 3579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:40.708850 3580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2033c6e9bc084ce3baa7be66008fcca6. Latest consensus state: current_term: 1 leader_uuid: "2033c6e9bc084ce3baa7be66008fcca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2033c6e9bc084ce3baa7be66008fcca6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44911 } } }
I20240501 07:59:40.708935 3580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6 [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:40.709151 3587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:59:40.711933 3587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:59:40.716856 3587 catalog_manager.cc:1347] Generated new cluster ID: 743d9b7daf1a4b899eef95d59516679e
I20240501 07:59:40.716933 3587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:59:40.731972 3587 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:59:40.733232 3587 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:59:40.740672 3587 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2033c6e9bc084ce3baa7be66008fcca6: Generated new TSK 0
I20240501 07:59:40.741580 3587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:59:40.753089 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44911
--builtin_ntp_servers=127.0.99.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 07:59:40.865543 3597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:40.865783 3597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:40.865828 3597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:40.869118 3597 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:40.869194 3597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:40.869278 3597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:59:40.873082 3597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44911
--never_fsync=true
--heap_profile_path=/tmp/kudu.3597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:40.874104 3597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:40.875299 3597 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:40.884338 3602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:40.885372 3605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:40.886153 3603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:40.887020 3597 server_base.cc:1034] running on GCE node
I20240501 07:59:40.887430 3597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:40.888024 3597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:40.889202 3597 hybrid_clock.cc:648] HybridClock initialized: now 1714550380889171 us; error 42 us; skew 500 ppm
I20240501 07:59:40.891389 3597 webserver.cc:460] Webserver started at http://127.0.99.129:42751/ using document root <none> and password file <none>
I20240501 07:59:40.891979 3597 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:40.892043 3597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:40.892243 3597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:40.893926 3597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
format_stamp: "Formatted at 2024-05-01 07:59:40 on dist-test-slave-wslg"
I20240501 07:59:40.894461 3597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
format_stamp: "Formatted at 2024-05-01 07:59:40 on dist-test-slave-wslg"
I20240501 07:59:40.897910 3597 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:40.900322 3611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:40.901407 3597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:40.901525 3597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
format_stamp: "Formatted at 2024-05-01 07:59:40 on dist-test-slave-wslg"
I20240501 07:59:40.901633 3597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:40.930743 3597 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:40.931471 3597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:40.931643 3597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:40.932188 3597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:40.933108 3597 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:40.933161 3597 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:40.933219 3597 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:40.933251 3597 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:40.941915 3597 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:42453
I20240501 07:59:40.941933 3724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:42453 every 8 connection(s)
I20240501 07:59:40.942909 3597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:59:40.949494 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 3597
I20240501 07:59:40.949607 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 07:59:40.952033 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44911
--builtin_ntp_servers=127.0.99.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:40.953399 3725 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44911
I20240501 07:59:40.953613 3725 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:40.954197 3725 heartbeater.cc:511] Master 127.0.99.190:44911 requested a full tablet report, sending...
I20240501 07:59:40.955956 3537 ts_manager.cc:194] Registered new tserver with Master: 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:40.957795 3537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:43157
W20240501 07:59:41.062182 3728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:41.062426 3728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:41.062475 3728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:41.065521 3728 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:41.065588 3728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:41.065655 3728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 07:59:41.069254 3728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44911
--never_fsync=true
--heap_profile_path=/tmp/kudu.3728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:41.070192 3728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:41.071333 3728 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:41.080411 3734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.081415 3733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.082412 3736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:41.083314 3728 server_base.cc:1034] running on GCE node
I20240501 07:59:41.083709 3728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:41.084301 3728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:41.085515 3728 hybrid_clock.cc:648] HybridClock initialized: now 1714550381085484 us; error 41 us; skew 500 ppm
I20240501 07:59:41.087672 3728 webserver.cc:460] Webserver started at http://127.0.99.130:42507/ using document root <none> and password file <none>
I20240501 07:59:41.088282 3728 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:41.088340 3728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:41.088536 3728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:41.090509 3728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:41.091017 3728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:41.094518 3728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 07:59:41.096829 3742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.097878 3728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:41.098016 3728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:41.098119 3728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:41.113643 3728 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:41.114363 3728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:41.114528 3728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:41.115060 3728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:41.115942 3728 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:41.115993 3728 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.116050 3728 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:41.116082 3728 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.124790 3728 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:43191
I20240501 07:59:41.124821 3855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:43191 every 8 connection(s)
I20240501 07:59:41.125819 3728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 07:59:41.128129 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 3728
I20240501 07:59:41.128260 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 07:59:41.130556 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44911
--builtin_ntp_servers=127.0.99.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:41.135902 3856 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44911
I20240501 07:59:41.136162 3856 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:41.136713 3856 heartbeater.cc:511] Master 127.0.99.190:44911 requested a full tablet report, sending...
I20240501 07:59:41.137741 3537 ts_manager.cc:194] Registered new tserver with Master: 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:41.138345 3537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:52439
W20240501 07:59:41.241242 3859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:41.241485 3859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:41.241557 3859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:41.244674 3859 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:41.244756 3859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:41.244851 3859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:59:41.248746 3859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44911
--never_fsync=true
--heap_profile_path=/tmp/kudu.3859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:41.249697 3859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:41.250856 3859 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:41.259685 3867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.260565 3864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.261766 3865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:41.262739 3859 server_base.cc:1034] running on GCE node
I20240501 07:59:41.263188 3859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:41.263788 3859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:41.264948 3859 hybrid_clock.cc:648] HybridClock initialized: now 1714550381264925 us; error 38 us; skew 500 ppm
I20240501 07:59:41.266948 3859 webserver.cc:460] Webserver started at http://127.0.99.131:43841/ using document root <none> and password file <none>
I20240501 07:59:41.267517 3859 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:41.267581 3859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:41.267764 3859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:41.269472 3859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8635309522a045b1941fea81699d524c"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:41.269968 3859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8635309522a045b1941fea81699d524c"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:41.273417 3859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 07:59:41.275717 3873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.276822 3859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:41.276957 3859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8635309522a045b1941fea81699d524c"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:41.277050 3859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:41.317430 3859 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:41.318187 3859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:41.318362 3859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:41.318917 3859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:41.319831 3859 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:41.319882 3859 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.319953 3859 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:41.319979 3859 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.328723 3859 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:39655
I20240501 07:59:41.328749 3986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:39655 every 8 connection(s)
I20240501 07:59:41.329694 3859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:59:41.337320 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 3859
I20240501 07:59:41.337427 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 07:59:41.338575 3987 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44911
I20240501 07:59:41.338810 3987 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:41.339347 3987 heartbeater.cc:511] Master 127.0.99.190:44911 requested a full tablet report, sending...
I20240501 07:59:41.339438 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44911
--builtin_ntp_servers=127.0.99.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:41.340384 3537 ts_manager.cc:194] Registered new tserver with Master: 8635309522a045b1941fea81699d524c (127.0.99.131:39655)
I20240501 07:59:41.340953 3537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:55733
W20240501 07:59:41.450824 3990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:41.451053 3990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:41.451097 3990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:41.454289 3990 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:41.454382 3990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:41.454484 3990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:59:41.458386 3990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44911
--never_fsync=true
--heap_profile_path=/tmp/kudu.3990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:41.459290 3990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:41.460430 3990 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:41.469678 3998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.470479 3995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.471277 3996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:41.472620 3990 server_base.cc:1034] running on GCE node
I20240501 07:59:41.473105 3990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:41.473747 3990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:41.474939 3990 hybrid_clock.cc:648] HybridClock initialized: now 1714550381474913 us; error 45 us; skew 500 ppm
I20240501 07:59:41.477051 3990 webserver.cc:460] Webserver started at http://127.0.99.132:40577/ using document root <none> and password file <none>
I20240501 07:59:41.477665 3990 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:41.477725 3990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:41.477958 3990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:41.479729 3990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "d8461c1ff1ed48b09047faeea87e9598"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:41.480239 3990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "d8461c1ff1ed48b09047faeea87e9598"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:41.483984 3990 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:59:41.486316 4004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.487371 3990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 07:59:41.487480 3990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d8461c1ff1ed48b09047faeea87e9598"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:41.487583 3990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:41.513366 3990 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:41.514065 3990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:41.514233 3990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:41.514783 3990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:41.515663 3990 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:41.515709 3990 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.515753 3990 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:41.515786 3990 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.524425 3990 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:37117
I20240501 07:59:41.524457 4117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:37117 every 8 connection(s)
I20240501 07:59:41.525410 3990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:59:41.525588 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 3990
I20240501 07:59:41.525686 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 07:59:41.528146 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44911
--builtin_ntp_servers=127.0.99.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:41.536119 4118 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44911
I20240501 07:59:41.536412 4118 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:41.537153 4118 heartbeater.cc:511] Master 127.0.99.190:44911 requested a full tablet report, sending...
I20240501 07:59:41.538350 3537 ts_manager.cc:194] Registered new tserver with Master: d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132:37117)
I20240501 07:59:41.538901 3537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:37659
W20240501 07:59:41.640134 4121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:41.640359 4121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:41.640404 4121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:41.643535 4121 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:41.643607 4121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:41.643679 4121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:59:41.647393 4121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44911
--never_fsync=true
--heap_profile_path=/tmp/kudu.4121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:41.648317 4121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:41.649452 4121 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:41.658628 4126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.659411 4129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.660311 4127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:41.661587 4121 server_base.cc:1034] running on GCE node
I20240501 07:59:41.662029 4121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:41.662670 4121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:41.663843 4121 hybrid_clock.cc:648] HybridClock initialized: now 1714550381663821 us; error 34 us; skew 500 ppm
I20240501 07:59:41.665891 4121 webserver.cc:460] Webserver started at http://127.0.99.133:45189/ using document root <none> and password file <none>
I20240501 07:59:41.666572 4121 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:41.666635 4121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:41.666826 4121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:41.668545 4121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "da103aa972ab49889fc0bce4abc56bf7"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:41.669037 4121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "da103aa972ab49889fc0bce4abc56bf7"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:41.672641 4121 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240501 07:59:41.675247 4135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.676481 4121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:41.676626 4121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "da103aa972ab49889fc0bce4abc56bf7"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:41.676738 4121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:41.688877 4121 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:41.689652 4121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:41.689837 4121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:41.690462 4121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:41.691520 4121 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:41.691578 4121 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.691628 4121 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:41.691658 4121 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.702256 4121 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:40225
I20240501 07:59:41.702309 4248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:40225 every 8 connection(s)
I20240501 07:59:41.703331 4121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:59:41.704682 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 4121
I20240501 07:59:41.704780 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 07:59:41.707381 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44911
--builtin_ntp_servers=127.0.99.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:41.713395 4249 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44911
I20240501 07:59:41.713649 4249 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:41.714277 4249 heartbeater.cc:511] Master 127.0.99.190:44911 requested a full tablet report, sending...
I20240501 07:59:41.715433 3537 ts_manager.cc:194] Registered new tserver with Master: da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133:40225)
I20240501 07:59:41.716267 3537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:46057
W20240501 07:59:41.823032 4252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:41.823263 4252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:41.823308 4252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:41.826583 4252 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:41.826661 4252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:41.826735 4252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:59:41.830431 4252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44911
--never_fsync=true
--heap_profile_path=/tmp/kudu.4252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:41.831420 4252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:41.832516 4252 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:41.841296 4258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.842064 4257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.842933 4260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:41.844262 4252 server_base.cc:1034] running on GCE node
I20240501 07:59:41.844702 4252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:41.845310 4252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:41.846472 4252 hybrid_clock.cc:648] HybridClock initialized: now 1714550381846464 us; error 30 us; skew 500 ppm
I20240501 07:59:41.848505 4252 webserver.cc:460] Webserver started at http://127.0.99.134:46473/ using document root <none> and password file <none>
I20240501 07:59:41.849099 4252 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:41.849184 4252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:41.849382 4252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:41.851094 4252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6208df88dcc34b4daf3d27a574e53a95"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:41.851573 4252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6208df88dcc34b4daf3d27a574e53a95"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:41.854941 4252 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:59:41.857224 4266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.858322 4252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:41.858445 4252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6208df88dcc34b4daf3d27a574e53a95"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:41.858537 4252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:41.877568 4252 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:41.878314 4252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:41.878473 4252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:41.879006 4252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:41.879909 4252 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:41.879959 4252 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.880023 4252 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:41.880051 4252 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.888862 4252 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:36011
I20240501 07:59:41.888885 4379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:36011 every 8 connection(s)
I20240501 07:59:41.889912 4252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:59:41.894137 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 4252
I20240501 07:59:41.894241 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 07:59:41.896144 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44911
--builtin_ntp_servers=127.0.99.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:41.898772 4380 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44911
I20240501 07:59:41.899032 4380 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:41.899731 4380 heartbeater.cc:511] Master 127.0.99.190:44911 requested a full tablet report, sending...
I20240501 07:59:41.900925 3537 ts_manager.cc:194] Registered new tserver with Master: 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134:36011)
I20240501 07:59:41.901482 3537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:54809
I20240501 07:59:41.960276 3725 heartbeater.cc:503] Master 127.0.99.190:44911 was elected leader, sending a full tablet report...
W20240501 07:59:42.008869 4383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:42.009151 4383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:42.009202 4383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:42.012408 4383 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:42.012485 4383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:42.012557 4383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:59:42.016307 4383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44911
--never_fsync=true
--heap_profile_path=/tmp/kudu.4383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:42.017241 4383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:42.018424 4383 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:42.027205 4388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:42.028237 4391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:42.029119 4389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:42.030040 4383 server_base.cc:1034] running on GCE node
I20240501 07:59:42.030491 4383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:42.031114 4383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:42.032272 4383 hybrid_clock.cc:648] HybridClock initialized: now 1714550382032246 us; error 41 us; skew 500 ppm
I20240501 07:59:42.034243 4383 webserver.cc:460] Webserver started at http://127.0.99.135:41587/ using document root <none> and password file <none>
I20240501 07:59:42.034806 4383 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:42.034868 4383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:42.035049 4383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:42.036717 4383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "09d8fa5c011b4e398612ba0274c14f2e"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-wslg"
I20240501 07:59:42.037187 4383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "09d8fa5c011b4e398612ba0274c14f2e"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-wslg"
I20240501 07:59:42.040519 4383 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240501 07:59:42.042758 4397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.043772 4383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240501 07:59:42.043884 4383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "09d8fa5c011b4e398612ba0274c14f2e"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-wslg"
I20240501 07:59:42.043970 4383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:42.055377 4383 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:42.056041 4383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:42.056208 4383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:42.056732 4383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:42.057628 4383 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:42.057679 4383 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.057740 4383 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:42.057771 4383 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.066277 4383 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:41559
I20240501 07:59:42.066327 4510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:41559 every 8 connection(s)
I20240501 07:59:42.067248 4383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:59:42.072168 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 4383
I20240501 07:59:42.072289 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 07:59:42.075951 4511 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44911
I20240501 07:59:42.076174 4511 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:42.076669 4511 heartbeater.cc:511] Master 127.0.99.190:44911 requested a full tablet report, sending...
I20240501 07:59:42.077587 3537 ts_manager.cc:194] Registered new tserver with Master: 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135:41559)
I20240501 07:59:42.078266 3537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:55937
I20240501 07:59:42.086558 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 07:59:42.095860 3537 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55978:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 07:59:42.113461 4445 tablet_service.cc:1467] Processing CreateTablet for tablet 71a2915d4efd4d48a1d41df5ecf2f4d7 (DEFAULT_TABLE table=TestTable [id=3af7e8ba491243f4b3f46209249cc0ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:42.114454 4445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a2915d4efd4d48a1d41df5ecf2f4d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:42.119741 4527 tablet_bootstrap.cc:492] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e: Bootstrap starting.
I20240501 07:59:42.119400 4052 tablet_service.cc:1467] Processing CreateTablet for tablet 71a2915d4efd4d48a1d41df5ecf2f4d7 (DEFAULT_TABLE table=TestTable [id=3af7e8ba491243f4b3f46209249cc0ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:42.120366 4052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a2915d4efd4d48a1d41df5ecf2f4d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:42.121727 4527 tablet_bootstrap.cc:654] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:42.122581 4527 log.cc:824] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:42.124253 4527 tablet_bootstrap.cc:492] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e: No bootstrap required, opened a new log
I20240501 07:59:42.124425 4527 ts_tablet_manager.cc:1367] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 07:59:42.125675 4529 tablet_bootstrap.cc:492] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598: Bootstrap starting.
I20240501 07:59:42.127743 4529 tablet_bootstrap.cc:654] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:42.128286 4527 raft_consensus.cc:348] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } }
I20240501 07:59:42.128618 4527 raft_consensus.cc:374] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:42.128695 4527 raft_consensus.cc:729] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d8fa5c011b4e398612ba0274c14f2e, State: Initialized, Role: FOLLOWER
I20240501 07:59:42.128880 4529 log.cc:824] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:42.129163 4527 consensus_queue.cc:260] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } }
I20240501 07:59:42.129798 4511 heartbeater.cc:503] Master 127.0.99.190:44911 was elected leader, sending a full tablet report...
I20240501 07:59:42.130008 4527 ts_tablet_manager.cc:1398] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240501 07:59:42.130643 4529 tablet_bootstrap.cc:492] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598: No bootstrap required, opened a new log
I20240501 07:59:42.130817 4529 ts_tablet_manager.cc:1367] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:59:42.133868 4529 raft_consensus.cc:348] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } }
I20240501 07:59:42.134124 4529 raft_consensus.cc:374] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:42.134202 4529 raft_consensus.cc:729] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8461c1ff1ed48b09047faeea87e9598, State: Initialized, Role: FOLLOWER
I20240501 07:59:42.134654 4529 consensus_queue.cc:260] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } }
I20240501 07:59:42.135363 4118 heartbeater.cc:503] Master 127.0.99.190:44911 was elected leader, sending a full tablet report...
I20240501 07:59:42.135548 4529 ts_tablet_manager.cc:1398] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:59:42.140254 3856 heartbeater.cc:503] Master 127.0.99.190:44911 was elected leader, sending a full tablet report...
I20240501 07:59:42.143396 4531 raft_consensus.cc:482] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:42.143607 4531 raft_consensus.cc:504] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } }
I20240501 07:59:42.144618 4531 leader_election.cc:288] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132:37117)
I20240501 07:59:42.148422 4072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a2915d4efd4d48a1d41df5ecf2f4d7" candidate_uuid: "09d8fa5c011b4e398612ba0274c14f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8461c1ff1ed48b09047faeea87e9598" is_pre_election: true
I20240501 07:59:42.148727 4072 raft_consensus.cc:2454] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09d8fa5c011b4e398612ba0274c14f2e in term 0.
I20240501 07:59:42.149242 4400 leader_election.cc:302] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 09d8fa5c011b4e398612ba0274c14f2e, d8461c1ff1ed48b09047faeea87e9598; no voters:
I20240501 07:59:42.149482 4531 raft_consensus.cc:2781] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:42.149551 4531 raft_consensus.cc:482] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:42.149582 4531 raft_consensus.cc:3037] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:42.150616 4531 raft_consensus.cc:504] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } }
I20240501 07:59:42.150916 4531 leader_election.cc:288] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [CANDIDATE]: Term 1 election: Requested vote from peers d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132:37117)
I20240501 07:59:42.151360 4072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a2915d4efd4d48a1d41df5ecf2f4d7" candidate_uuid: "09d8fa5c011b4e398612ba0274c14f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8461c1ff1ed48b09047faeea87e9598"
I20240501 07:59:42.151477 4072 raft_consensus.cc:3037] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:42.152374 4072 raft_consensus.cc:2454] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09d8fa5c011b4e398612ba0274c14f2e in term 1.
I20240501 07:59:42.152721 4400 leader_election.cc:302] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 09d8fa5c011b4e398612ba0274c14f2e, d8461c1ff1ed48b09047faeea87e9598; no voters:
I20240501 07:59:42.152876 4531 raft_consensus.cc:2781] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:42.153090 4531 raft_consensus.cc:686] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 LEADER]: Becoming Leader. State: Replica: 09d8fa5c011b4e398612ba0274c14f2e, State: Running, Role: LEADER
I20240501 07:59:42.153422 4531 consensus_queue.cc:237] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } }
I20240501 07:59:42.156596 3537 catalog_manager.cc:5484] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135). New cstate: current_term: 1 leader_uuid: "09d8fa5c011b4e398612ba0274c14f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:42.170135 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 07:59:42.171483 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 014bc50d0c5a44bdbf49c0eb41fa93ad to finish bootstrapping
I20240501 07:59:42.177466 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 40c9ccd5f29c424eb845d670a6f7d21e to finish bootstrapping
I20240501 07:59:42.183459 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8635309522a045b1941fea81699d524c to finish bootstrapping
I20240501 07:59:42.189210 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d8461c1ff1ed48b09047faeea87e9598 to finish bootstrapping
I20240501 07:59:42.194378 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver da103aa972ab49889fc0bce4abc56bf7 to finish bootstrapping
I20240501 07:59:42.200417 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6208df88dcc34b4daf3d27a574e53a95 to finish bootstrapping
I20240501 07:59:42.206656 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 09d8fa5c011b4e398612ba0274c14f2e to finish bootstrapping
I20240501 07:59:42.211993 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 3990
I20240501 07:59:42.218757 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 4121
I20240501 07:59:42.224303 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 4252
I20240501 07:59:42.230276 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 4383
I20240501 07:59:42.343077 3987 heartbeater.cc:503] Master 127.0.99.190:44911 was elected leader, sending a full tablet report...
I20240501 07:59:45.987322 3537 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55978:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:46.000705 3920 tablet_service.cc:1467] Processing CreateTablet for tablet 72c5108aa1aa4cbd913d9ceb1a262d58 (DEFAULT_TABLE table=rebalance_test_table_0 [id=294054b1ea344845ad358beee1ef488a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.000844 3659 tablet_service.cc:1467] Processing CreateTablet for tablet f1aa85993dc449b7b33ebccb4e907046 (DEFAULT_TABLE table=rebalance_test_table_0 [id=294054b1ea344845ad358beee1ef488a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.000844 3790 tablet_service.cc:1467] Processing CreateTablet for tablet f1aa85993dc449b7b33ebccb4e907046 (DEFAULT_TABLE table=rebalance_test_table_0 [id=294054b1ea344845ad358beee1ef488a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.001787 3790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1aa85993dc449b7b33ebccb4e907046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.001787 3659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1aa85993dc449b7b33ebccb4e907046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.001803 3920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c5108aa1aa4cbd913d9ceb1a262d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.001119 3658 tablet_service.cc:1467] Processing CreateTablet for tablet 02f63ed2bd6345b686c74eb002bb306f (DEFAULT_TABLE table=rebalance_test_table_0 [id=294054b1ea344845ad358beee1ef488a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.002118 3658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02f63ed2bd6345b686c74eb002bb306f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.003326 3789 tablet_service.cc:1467] Processing CreateTablet for tablet 72c5108aa1aa4cbd913d9ceb1a262d58 (DEFAULT_TABLE table=rebalance_test_table_0 [id=294054b1ea344845ad358beee1ef488a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.003654 3789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c5108aa1aa4cbd913d9ceb1a262d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.000708 3921 tablet_service.cc:1467] Processing CreateTablet for tablet 02f63ed2bd6345b686c74eb002bb306f (DEFAULT_TABLE table=rebalance_test_table_0 [id=294054b1ea344845ad358beee1ef488a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.004779 3921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02f63ed2bd6345b686c74eb002bb306f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.008000 4551 tablet_bootstrap.cc:492] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 40c9ccd5f29c424eb845d670a6f7d21e: Bootstrap starting.
I20240501 07:59:46.010483 4551 tablet_bootstrap.cc:654] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 40c9ccd5f29c424eb845d670a6f7d21e: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.010663 4553 tablet_bootstrap.cc:492] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c: Bootstrap starting.
I20240501 07:59:46.011217 4551 log.cc:824] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 40c9ccd5f29c424eb845d670a6f7d21e: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:46.011726 4552 tablet_bootstrap.cc:492] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad: Bootstrap starting.
I20240501 07:59:46.012727 4553 tablet_bootstrap.cc:654] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.013495 4553 log.cc:824] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:46.013676 4552 tablet_bootstrap.cc:654] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.014452 4552 log.cc:824] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:46.015275 4553 tablet_bootstrap.cc:492] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c: No bootstrap required, opened a new log
I20240501 07:59:46.015393 4551 tablet_bootstrap.cc:492] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 40c9ccd5f29c424eb845d670a6f7d21e: No bootstrap required, opened a new log
I20240501 07:59:46.015483 4553 ts_tablet_manager.cc:1367] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:59:46.015581 4551 ts_tablet_manager.cc:1367] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 07:59:46.016160 4552 tablet_bootstrap.cc:492] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad: No bootstrap required, opened a new log
I20240501 07:59:46.016324 4552 ts_tablet_manager.cc:1367] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 07:59:46.018384 4553 raft_consensus.cc:348] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.018565 4553 raft_consensus.cc:374] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.018623 4553 raft_consensus.cc:729] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8635309522a045b1941fea81699d524c, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.019095 4553 consensus_queue.cc:260] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.019409 4551 raft_consensus.cc:348] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.019625 4551 raft_consensus.cc:374] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.019695 4551 raft_consensus.cc:729] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.020010 4553 ts_tablet_manager.cc:1398] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 07:59:46.019982 4552 raft_consensus.cc:348] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.020164 4552 raft_consensus.cc:374] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.020170 4551 consensus_queue.cc:260] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 40c9ccd5f29c424eb845d670a6f7d21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.020364 4553 tablet_bootstrap.cc:492] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c: Bootstrap starting.
I20240501 07:59:46.020509 4552 raft_consensus.cc:729] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.020999 4551 ts_tablet_manager.cc:1398] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:59:46.020982 4552 consensus_queue.cc:260] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.021709 4553 tablet_bootstrap.cc:654] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.021839 4552 ts_tablet_manager.cc:1398] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 07:59:46.022246 4551 tablet_bootstrap.cc:492] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e: Bootstrap starting.
I20240501 07:59:46.022249 4552 tablet_bootstrap.cc:492] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Bootstrap starting.
I20240501 07:59:46.023190 4552 tablet_bootstrap.cc:654] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.023190 4551 tablet_bootstrap.cc:654] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.024454 4552 tablet_bootstrap.cc:492] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad: No bootstrap required, opened a new log
I20240501 07:59:46.024572 4552 ts_tablet_manager.cc:1367] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:46.025238 4551 tablet_bootstrap.cc:492] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e: No bootstrap required, opened a new log
I20240501 07:59:46.025202 4552 raft_consensus.cc:348] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:46.025311 4552 raft_consensus.cc:374] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.025323 4551 ts_tablet_manager.cc:1367] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:46.025346 4552 raft_consensus.cc:729] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.025477 4552 consensus_queue.cc:260] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:46.025759 4552 ts_tablet_manager.cc:1398] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:46.025918 4551 raft_consensus.cc:348] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:46.026095 4551 raft_consensus.cc:374] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.026144 4551 raft_consensus.cc:729] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.026330 4551 consensus_queue.cc:260] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:46.026590 4551 ts_tablet_manager.cc:1398] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:46.026846 4553 tablet_bootstrap.cc:492] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c: No bootstrap required, opened a new log
I20240501 07:59:46.026962 4553 ts_tablet_manager.cc:1367] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:46.027504 4553 raft_consensus.cc:348] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.027630 4553 raft_consensus.cc:374] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.027670 4553 raft_consensus.cc:729] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8635309522a045b1941fea81699d524c, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.027797 4553 consensus_queue.cc:260] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.028076 4553 ts_tablet_manager.cc:1398] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:46.051388 4559 raft_consensus.cc:482] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:46.051589 4559 raft_consensus.cc:504] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:46.052634 4559 leader_election.cc:288] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:46.056265 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1aa85993dc449b7b33ebccb4e907046" candidate_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" is_pre_election: true
I20240501 07:59:46.056583 3810 raft_consensus.cc:2454] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014bc50d0c5a44bdbf49c0eb41fa93ad in term 0.
I20240501 07:59:46.057042 3614 leader_election.cc:302] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 40c9ccd5f29c424eb845d670a6f7d21e; no voters:
I20240501 07:59:46.057330 4559 raft_consensus.cc:2781] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:46.057404 4559 raft_consensus.cc:482] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:46.057433 4559 raft_consensus.cc:3037] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:46.058363 4559 raft_consensus.cc:504] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:46.058657 4559 leader_election.cc:288] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 1 election: Requested vote from peers 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:46.059072 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1aa85993dc449b7b33ebccb4e907046" candidate_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
I20240501 07:59:46.059207 3810 raft_consensus.cc:3037] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:46.060077 3810 raft_consensus.cc:2454] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014bc50d0c5a44bdbf49c0eb41fa93ad in term 1.
I20240501 07:59:46.060365 3614 leader_election.cc:302] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 40c9ccd5f29c424eb845d670a6f7d21e; no voters:
I20240501 07:59:46.060528 4559 raft_consensus.cc:2781] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:46.060724 4559 raft_consensus.cc:686] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Becoming Leader. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Running, Role: LEADER
I20240501 07:59:46.061041 4559 consensus_queue.cc:237] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:46.064231 3535 catalog_manager.cc:5484] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129). New cstate: current_term: 1 leader_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:46.084057 3988 tablet.cc:2369] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:46.087311 4557 raft_consensus.cc:482] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:46.087519 4557 raft_consensus.cc:504] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.088627 4557 leader_election.cc:288] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:46.092216 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c5108aa1aa4cbd913d9ceb1a262d58" candidate_uuid: "8635309522a045b1941fea81699d524c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" is_pre_election: true
I20240501 07:59:46.092357 3810 raft_consensus.cc:2454] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8635309522a045b1941fea81699d524c in term 0.
I20240501 07:59:46.092679 3876 leader_election.cc:302] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 40c9ccd5f29c424eb845d670a6f7d21e, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:46.092948 4557 raft_consensus.cc:2781] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:46.093021 4557 raft_consensus.cc:482] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:46.093078 4557 raft_consensus.cc:3037] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:46.093952 4557 raft_consensus.cc:504] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.094259 4557 leader_election.cc:288] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 election: Requested vote from peers 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:46.094668 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c5108aa1aa4cbd913d9ceb1a262d58" candidate_uuid: "8635309522a045b1941fea81699d524c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
I20240501 07:59:46.094791 3810 raft_consensus.cc:3037] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:46.095506 3810 raft_consensus.cc:2454] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8635309522a045b1941fea81699d524c in term 1.
I20240501 07:59:46.095763 3876 leader_election.cc:302] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 40c9ccd5f29c424eb845d670a6f7d21e, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:46.095925 4557 raft_consensus.cc:2781] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:46.096149 4557 raft_consensus.cc:686] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Becoming Leader. State: Replica: 8635309522a045b1941fea81699d524c, State: Running, Role: LEADER
I20240501 07:59:46.096496 4557 consensus_queue.cc:237] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.099399 3535 catalog_manager.cc:5484] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8635309522a045b1941fea81699d524c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } }
W20240501 07:59:46.130852 3857 tablet.cc:2369] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 40c9ccd5f29c424eb845d670a6f7d21e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:46.135526 4559 raft_consensus.cc:482] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:46.135675 4559 raft_consensus.cc:504] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.136104 4559 leader_election.cc:288] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8635309522a045b1941fea81699d524c (127.0.99.131:39655)
I20240501 07:59:46.139626 3941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f63ed2bd6345b686c74eb002bb306f" candidate_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8635309522a045b1941fea81699d524c" is_pre_election: true
I20240501 07:59:46.139823 3941 raft_consensus.cc:2454] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014bc50d0c5a44bdbf49c0eb41fa93ad in term 0.
I20240501 07:59:46.140179 3614 leader_election.cc:302] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:46.140345 4559 raft_consensus.cc:2781] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:46.140401 4559 raft_consensus.cc:482] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:46.140437 4559 raft_consensus.cc:3037] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:46.141223 4559 raft_consensus.cc:504] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.141485 4559 leader_election.cc:288] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 1 election: Requested vote from peers 8635309522a045b1941fea81699d524c (127.0.99.131:39655)
I20240501 07:59:46.141847 3941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f63ed2bd6345b686c74eb002bb306f" candidate_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8635309522a045b1941fea81699d524c"
I20240501 07:59:46.141955 3941 raft_consensus.cc:3037] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:46.142725 3941 raft_consensus.cc:2454] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014bc50d0c5a44bdbf49c0eb41fa93ad in term 1.
I20240501 07:59:46.142989 3614 leader_election.cc:302] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:46.143154 4559 raft_consensus.cc:2781] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:46.143270 4559 raft_consensus.cc:686] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Becoming Leader. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Running, Role: LEADER
I20240501 07:59:46.143416 4559 consensus_queue.cc:237] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.144953 3535 catalog_manager.cc:5484] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129). New cstate: current_term: 1 leader_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: HEALTHY } } }
W20240501 07:59:46.197700 3726 tablet.cc:2369] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:59:46.260758 4570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:46.261042 4570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:46.264232 4570 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 07:59:46.362458 3941 raft_consensus.cc:1261] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:46.362496 3810 raft_consensus.cc:1261] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:46.363229 4559 consensus_queue.cc:1035] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:46.366650 4559 consensus_queue.cc:1035] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:46.372679 3809 raft_consensus.cc:1261] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:46.373179 4557 consensus_queue.cc:1035] T 72c5108aa1aa4cbd913d9ceb1a262d58 P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:46.375720 4597 mvcc.cc:205] Tried to move back new op lower bound from 7022798382548848640 to 7022798381644468224. Current Snapshot: MvccSnapshot[applied={T|T < 7022798382533009408}]
I20240501 07:59:46.378525 4602 mvcc.cc:205] Tried to move back new op lower bound from 7022798382548848640 to 7022798381644468224. Current Snapshot: MvccSnapshot[applied={T|T < 7022798382533009408}]
I20240501 07:59:46.381374 4600 mvcc.cc:205] Tried to move back new op lower bound from 7022798382533050368 to 7022798381308567552. Current Snapshot: MvccSnapshot[applied={T|T < 7022798382533050368}]
INSERT report
rows total: 2000
time total: 110.587 ms
time per row: 0.0552934 ms
I20240501 07:59:46.422531 3534 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55978:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:46.428515 3658 tablet_service.cc:1467] Processing CreateTablet for tablet a77d5e3c352f495cbee97be6a2c4661a (DEFAULT_TABLE table=rebalance_test_table_1 [id=4c0194cc5d4f4b1587cce57551ae4de8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.428532 3790 tablet_service.cc:1467] Processing CreateTablet for tablet 5432d22bf6dc4866bbc3d91f21a45c21 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4c0194cc5d4f4b1587cce57551ae4de8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.428542 3789 tablet_service.cc:1467] Processing CreateTablet for tablet f75acaf04c12462aa4b3fa7e019776d4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4c0194cc5d4f4b1587cce57551ae4de8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.428866 3658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a77d5e3c352f495cbee97be6a2c4661a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.428867 3790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5432d22bf6dc4866bbc3d91f21a45c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.428814 3920 tablet_service.cc:1467] Processing CreateTablet for tablet a77d5e3c352f495cbee97be6a2c4661a (DEFAULT_TABLE table=rebalance_test_table_1 [id=4c0194cc5d4f4b1587cce57551ae4de8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.429059 3789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75acaf04c12462aa4b3fa7e019776d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.429077 3920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a77d5e3c352f495cbee97be6a2c4661a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.431357 4552 tablet_bootstrap.cc:492] T a77d5e3c352f495cbee97be6a2c4661a P 014bc50d0c5a44bdbf49c0eb41fa93ad: Bootstrap starting.
I20240501 07:59:46.431488 3921 tablet_service.cc:1467] Processing CreateTablet for tablet f75acaf04c12462aa4b3fa7e019776d4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4c0194cc5d4f4b1587cce57551ae4de8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.431744 3921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75acaf04c12462aa4b3fa7e019776d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.431802 4553 tablet_bootstrap.cc:492] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c: Bootstrap starting.
I20240501 07:59:46.431870 4551 tablet_bootstrap.cc:492] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e: Bootstrap starting.
I20240501 07:59:46.432299 4552 tablet_bootstrap.cc:654] T a77d5e3c352f495cbee97be6a2c4661a P 014bc50d0c5a44bdbf49c0eb41fa93ad: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.432713 4553 tablet_bootstrap.cc:654] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.432785 4551 tablet_bootstrap.cc:654] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.428515 3659 tablet_service.cc:1467] Processing CreateTablet for tablet 5432d22bf6dc4866bbc3d91f21a45c21 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4c0194cc5d4f4b1587cce57551ae4de8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.433789 3659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5432d22bf6dc4866bbc3d91f21a45c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.433945 4552 tablet_bootstrap.cc:492] T a77d5e3c352f495cbee97be6a2c4661a P 014bc50d0c5a44bdbf49c0eb41fa93ad: No bootstrap required, opened a new log
I20240501 07:59:46.434064 4552 ts_tablet_manager.cc:1367] T a77d5e3c352f495cbee97be6a2c4661a P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:46.434427 4553 tablet_bootstrap.cc:492] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c: No bootstrap required, opened a new log
I20240501 07:59:46.434511 4553 ts_tablet_manager.cc:1367] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:46.434574 4552 raft_consensus.cc:348] T a77d5e3c352f495cbee97be6a2c4661a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.434679 4552 raft_consensus.cc:374] T a77d5e3c352f495cbee97be6a2c4661a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.434707 4552 raft_consensus.cc:729] T a77d5e3c352f495cbee97be6a2c4661a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.434829 4552 consensus_queue.cc:260] T a77d5e3c352f495cbee97be6a2c4661a P 014bc50d0c5a44bdbf49c0eb41fa93ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.434911 4551 tablet_bootstrap.cc:492] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e: No bootstrap required, opened a new log
I20240501 07:59:46.434998 4551 ts_tablet_manager.cc:1367] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:46.435277 4552 ts_tablet_manager.cc:1398] T a77d5e3c352f495cbee97be6a2c4661a P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:46.435469 4551 raft_consensus.cc:348] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.435562 4551 raft_consensus.cc:374] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.435592 4551 raft_consensus.cc:729] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.435549 4553 raft_consensus.cc:348] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.435686 4553 raft_consensus.cc:374] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.435725 4553 raft_consensus.cc:729] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8635309522a045b1941fea81699d524c, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.435715 4551 consensus_queue.cc:260] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.435837 4553 consensus_queue.cc:260] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.435981 4551 ts_tablet_manager.cc:1398] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:46.436089 4553 ts_tablet_manager.cc:1398] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:46.436153 4551 tablet_bootstrap.cc:492] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e: Bootstrap starting.
I20240501 07:59:46.436239 4553 tablet_bootstrap.cc:492] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c: Bootstrap starting.
I20240501 07:59:46.436408 4552 tablet_bootstrap.cc:492] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Bootstrap starting.
I20240501 07:59:46.437129 4551 tablet_bootstrap.cc:654] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.437423 4553 tablet_bootstrap.cc:654] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.438670 4551 tablet_bootstrap.cc:492] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e: No bootstrap required, opened a new log
I20240501 07:59:46.438717 4552 tablet_bootstrap.cc:654] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.438817 4551 ts_tablet_manager.cc:1367] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:46.439639 4551 raft_consensus.cc:348] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.439818 4551 raft_consensus.cc:374] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.439899 4551 raft_consensus.cc:729] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.440132 4552 tablet_bootstrap.cc:492] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad: No bootstrap required, opened a new log
I20240501 07:59:46.440093 4551 consensus_queue.cc:260] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.440222 4552 ts_tablet_manager.cc:1367] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:46.440482 4553 tablet_bootstrap.cc:492] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c: No bootstrap required, opened a new log
I20240501 07:59:46.440572 4553 ts_tablet_manager.cc:1367] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:46.440708 4551 ts_tablet_manager.cc:1398] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:46.440946 4552 raft_consensus.cc:348] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.441048 4552 raft_consensus.cc:374] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.441016 4553 raft_consensus.cc:348] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.441084 4552 raft_consensus.cc:729] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.441119 4553 raft_consensus.cc:374] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.441171 4553 raft_consensus.cc:729] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8635309522a045b1941fea81699d524c, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.441205 4552 consensus_queue.cc:260] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.441291 4553 consensus_queue.cc:260] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.441498 4552 ts_tablet_manager.cc:1398] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:46.441571 4553 ts_tablet_manager.cc:1398] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:46.452137 4558 raft_consensus.cc:482] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:46.452282 4558 raft_consensus.cc:504] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.453173 4558 leader_election.cc:288] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:46.456888 3679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5432d22bf6dc4866bbc3d91f21a45c21" candidate_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" is_pre_election: true
I20240501 07:59:46.457124 3679 raft_consensus.cc:2454] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c9ccd5f29c424eb845d670a6f7d21e in term 0.
I20240501 07:59:46.457551 3745 leader_election.cc:302] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 40c9ccd5f29c424eb845d670a6f7d21e; no voters:
I20240501 07:59:46.457784 4558 raft_consensus.cc:2781] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:46.457844 4558 raft_consensus.cc:482] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:46.457887 4558 raft_consensus.cc:3037] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:46.458773 4558 raft_consensus.cc:504] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.459050 4558 leader_election.cc:288] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 1 election: Requested vote from peers 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:46.459419 3679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5432d22bf6dc4866bbc3d91f21a45c21" candidate_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
I20240501 07:59:46.459525 3679 raft_consensus.cc:3037] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:46.460340 3679 raft_consensus.cc:2454] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c9ccd5f29c424eb845d670a6f7d21e in term 1.
I20240501 07:59:46.460610 3745 leader_election.cc:302] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 40c9ccd5f29c424eb845d670a6f7d21e; no voters:
I20240501 07:59:46.460767 4558 raft_consensus.cc:2781] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:46.460961 4558 raft_consensus.cc:686] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Becoming Leader. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Running, Role: LEADER
I20240501 07:59:46.461246 4558 consensus_queue.cc:237] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.463325 3533 catalog_manager.cc:5484] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130). New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:46.494868 4557 raft_consensus.cc:482] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:46.495034 4557 raft_consensus.cc:504] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.495539 4557 leader_election.cc:288] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:46.499101 3679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a77d5e3c352f495cbee97be6a2c4661a" candidate_uuid: "8635309522a045b1941fea81699d524c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" is_pre_election: true
I20240501 07:59:46.499293 3679 raft_consensus.cc:2454] T a77d5e3c352f495cbee97be6a2c4661a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8635309522a045b1941fea81699d524c in term 0.
I20240501 07:59:46.499601 3876 leader_election.cc:302] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:46.499823 4557 raft_consensus.cc:2781] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:46.499888 4557 raft_consensus.cc:482] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:46.499920 4557 raft_consensus.cc:3037] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:46.500780 4557 raft_consensus.cc:504] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.501088 4557 leader_election.cc:288] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 election: Requested vote from peers 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:46.501546 3679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a77d5e3c352f495cbee97be6a2c4661a" candidate_uuid: "8635309522a045b1941fea81699d524c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
I20240501 07:59:46.501667 3679 raft_consensus.cc:3037] T a77d5e3c352f495cbee97be6a2c4661a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:46.502502 3679 raft_consensus.cc:2454] T a77d5e3c352f495cbee97be6a2c4661a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8635309522a045b1941fea81699d524c in term 1.
I20240501 07:59:46.502813 3876 leader_election.cc:302] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:46.503034 4557 raft_consensus.cc:2781] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:46.503149 4557 raft_consensus.cc:686] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Becoming Leader. State: Replica: 8635309522a045b1941fea81699d524c, State: Running, Role: LEADER
I20240501 07:59:46.503304 4557 consensus_queue.cc:237] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.504912 3533 catalog_manager.cc:5484] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8635309522a045b1941fea81699d524c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:46.548063 4558 raft_consensus.cc:482] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:46.548228 4558 raft_consensus.cc:504] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.548684 4558 leader_election.cc:288] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8635309522a045b1941fea81699d524c (127.0.99.131:39655)
I20240501 07:59:46.551985 3941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75acaf04c12462aa4b3fa7e019776d4" candidate_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8635309522a045b1941fea81699d524c" is_pre_election: true
I20240501 07:59:46.552140 3941 raft_consensus.cc:2454] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c9ccd5f29c424eb845d670a6f7d21e in term 0.
I20240501 07:59:46.552448 3745 leader_election.cc:302] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 40c9ccd5f29c424eb845d670a6f7d21e, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:46.552635 4558 raft_consensus.cc:2781] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:46.552703 4558 raft_consensus.cc:482] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:46.552752 4558 raft_consensus.cc:3037] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:46.553592 4558 raft_consensus.cc:504] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.553891 4558 leader_election.cc:288] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 1 election: Requested vote from peers 8635309522a045b1941fea81699d524c (127.0.99.131:39655)
I20240501 07:59:46.554376 3941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75acaf04c12462aa4b3fa7e019776d4" candidate_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8635309522a045b1941fea81699d524c"
I20240501 07:59:46.554512 3941 raft_consensus.cc:3037] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:46.555310 3941 raft_consensus.cc:2454] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c9ccd5f29c424eb845d670a6f7d21e in term 1.
I20240501 07:59:46.555590 3745 leader_election.cc:302] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 40c9ccd5f29c424eb845d670a6f7d21e, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:46.555754 4558 raft_consensus.cc:2781] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:46.555843 4558 raft_consensus.cc:686] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Becoming Leader. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Running, Role: LEADER
I20240501 07:59:46.555991 4558 consensus_queue.cc:237] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.557538 3533 catalog_manager.cc:5484] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130). New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:46.695698 4627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:46.696082 4627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:46.699362 4627 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 07:59:46.780550 3679 raft_consensus.cc:1261] T a77d5e3c352f495cbee97be6a2c4661a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:46.781617 3941 raft_consensus.cc:1261] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:46.782398 4623 consensus_queue.cc:1035] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:46.782891 4557 consensus_queue.cc:1035] T a77d5e3c352f495cbee97be6a2c4661a P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:46.784765 3679 raft_consensus.cc:1261] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:46.785818 4558 consensus_queue.cc:1035] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 86.6341 ms
time per row: 0.043317 ms
I20240501 07:59:46.829521 3533 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55978:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:46.835251 3659 tablet_service.cc:1467] Processing CreateTablet for tablet 7c0cda2d32534bb29a6a2538ac13368a (DEFAULT_TABLE table=rebalance_test_table_2 [id=712e7be85ad54a6ebf02a160dca63546]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.835439 3789 tablet_service.cc:1467] Processing CreateTablet for tablet 7c0cda2d32534bb29a6a2538ac13368a (DEFAULT_TABLE table=rebalance_test_table_2 [id=712e7be85ad54a6ebf02a160dca63546]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.835714 3789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0cda2d32534bb29a6a2538ac13368a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.835597 3659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0cda2d32534bb29a6a2538ac13368a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.835597 3921 tablet_service.cc:1467] Processing CreateTablet for tablet cc31e13bb07f44ddb7b80282ee4c434b (DEFAULT_TABLE table=rebalance_test_table_2 [id=712e7be85ad54a6ebf02a160dca63546]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.836488 3921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc31e13bb07f44ddb7b80282ee4c434b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.837787 4552 tablet_bootstrap.cc:492] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad: Bootstrap starting.
I20240501 07:59:46.838537 4551 tablet_bootstrap.cc:492] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e: Bootstrap starting.
I20240501 07:59:46.838797 4552 tablet_bootstrap.cc:654] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.838780 3920 tablet_service.cc:1467] Processing CreateTablet for tablet ed30e32a4ba2483484b5bc5e3cfb44fd (DEFAULT_TABLE table=rebalance_test_table_2 [id=712e7be85ad54a6ebf02a160dca63546]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.839000 4553 tablet_bootstrap.cc:492] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c: Bootstrap starting.
I20240501 07:59:46.839004 3920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed30e32a4ba2483484b5bc5e3cfb44fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.839259 3789 tablet_service.cc:1467] Processing CreateTablet for tablet ed30e32a4ba2483484b5bc5e3cfb44fd (DEFAULT_TABLE table=rebalance_test_table_2 [id=712e7be85ad54a6ebf02a160dca63546]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.839433 4551 tablet_bootstrap.cc:654] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.839485 3789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed30e32a4ba2483484b5bc5e3cfb44fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.840375 3659 tablet_service.cc:1467] Processing CreateTablet for tablet cc31e13bb07f44ddb7b80282ee4c434b (DEFAULT_TABLE table=rebalance_test_table_2 [id=712e7be85ad54a6ebf02a160dca63546]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:46.840617 3659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc31e13bb07f44ddb7b80282ee4c434b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:46.841791 4553 tablet_bootstrap.cc:654] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.842119 4552 tablet_bootstrap.cc:492] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad: No bootstrap required, opened a new log
I20240501 07:59:46.842283 4552 ts_tablet_manager.cc:1367] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:46.842485 4551 tablet_bootstrap.cc:492] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e: No bootstrap required, opened a new log
I20240501 07:59:46.842568 4551 ts_tablet_manager.cc:1367] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:46.842726 4552 raft_consensus.cc:348] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:46.842829 4552 raft_consensus.cc:374] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.842864 4552 raft_consensus.cc:729] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.842996 4552 consensus_queue.cc:260] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:46.843070 4553 tablet_bootstrap.cc:492] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c: No bootstrap required, opened a new log
I20240501 07:59:46.843050 4551 raft_consensus.cc:348] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:46.843137 4553 ts_tablet_manager.cc:1367] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:59:46.843164 4551 raft_consensus.cc:374] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.843206 4551 raft_consensus.cc:729] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.843293 4552 ts_tablet_manager.cc:1398] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:46.843328 4551 consensus_queue.cc:260] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:46.843464 4552 tablet_bootstrap.cc:492] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad: Bootstrap starting.
I20240501 07:59:46.843595 4551 ts_tablet_manager.cc:1398] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:46.843550 4553 raft_consensus.cc:348] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.843642 4553 raft_consensus.cc:374] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.843673 4553 raft_consensus.cc:729] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8635309522a045b1941fea81699d524c, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.843770 4551 tablet_bootstrap.cc:492] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e: Bootstrap starting.
I20240501 07:59:46.843792 4553 consensus_queue.cc:260] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.844040 4553 ts_tablet_manager.cc:1398] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:46.844220 4553 tablet_bootstrap.cc:492] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c: Bootstrap starting.
I20240501 07:59:46.844343 4552 tablet_bootstrap.cc:654] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.844868 4551 tablet_bootstrap.cc:654] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.845070 4553 tablet_bootstrap.cc:654] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:46.845435 4552 tablet_bootstrap.cc:492] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad: No bootstrap required, opened a new log
I20240501 07:59:46.845505 4552 ts_tablet_manager.cc:1367] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:46.845942 4552 raft_consensus.cc:348] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.846125 4552 raft_consensus.cc:374] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.846179 4552 raft_consensus.cc:729] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.846298 4552 consensus_queue.cc:260] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:46.846539 4552 ts_tablet_manager.cc:1398] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:46.846851 4551 tablet_bootstrap.cc:492] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e: No bootstrap required, opened a new log
I20240501 07:59:46.846956 4551 ts_tablet_manager.cc:1367] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:46.847198 4553 tablet_bootstrap.cc:492] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c: No bootstrap required, opened a new log
I20240501 07:59:46.847275 4553 ts_tablet_manager.cc:1367] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:59:46.847362 4551 raft_consensus.cc:348] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.847514 4551 raft_consensus.cc:374] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.847553 4551 raft_consensus.cc:729] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.847676 4551 consensus_queue.cc:260] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.848157 4553 raft_consensus.cc:348] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.848266 4553 raft_consensus.cc:374] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:46.848307 4553 raft_consensus.cc:729] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8635309522a045b1941fea81699d524c, State: Initialized, Role: FOLLOWER
I20240501 07:59:46.848456 4551 ts_tablet_manager.cc:1398] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:46.848440 4553 consensus_queue.cc:260] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.848773 4553 ts_tablet_manager.cc:1398] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:46.894994 4623 raft_consensus.cc:482] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:46.895126 4623 raft_consensus.cc:504] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.895483 4623 leader_election.cc:288] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8635309522a045b1941fea81699d524c (127.0.99.131:39655)
I20240501 07:59:46.895947 3941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed30e32a4ba2483484b5bc5e3cfb44fd" candidate_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8635309522a045b1941fea81699d524c" is_pre_election: true
I20240501 07:59:46.896092 3941 raft_consensus.cc:2454] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c9ccd5f29c424eb845d670a6f7d21e in term 0.
I20240501 07:59:46.896421 3745 leader_election.cc:302] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 40c9ccd5f29c424eb845d670a6f7d21e, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:46.896587 4623 raft_consensus.cc:2781] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:46.896647 4623 raft_consensus.cc:482] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:46.896684 4623 raft_consensus.cc:3037] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:46.897540 4623 raft_consensus.cc:504] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.897794 4623 leader_election.cc:288] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 1 election: Requested vote from peers 8635309522a045b1941fea81699d524c (127.0.99.131:39655)
I20240501 07:59:46.898219 3941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed30e32a4ba2483484b5bc5e3cfb44fd" candidate_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8635309522a045b1941fea81699d524c"
I20240501 07:59:46.898351 3941 raft_consensus.cc:3037] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:46.899045 3941 raft_consensus.cc:2454] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c9ccd5f29c424eb845d670a6f7d21e in term 1.
I20240501 07:59:46.899346 3745 leader_election.cc:302] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 40c9ccd5f29c424eb845d670a6f7d21e, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:46.899498 4623 raft_consensus.cc:2781] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:46.899585 4623 raft_consensus.cc:686] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Becoming Leader. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Running, Role: LEADER
I20240501 07:59:46.899765 4623 consensus_queue.cc:237] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:46.901389 3532 catalog_manager.cc:5484] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130). New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:47.083546 4564 raft_consensus.cc:482] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:47.083696 4564 raft_consensus.cc:504] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:47.084048 4564 leader_election.cc:288] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:47.084580 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0cda2d32534bb29a6a2538ac13368a" candidate_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" is_pre_election: true
I20240501 07:59:47.084729 3809 raft_consensus.cc:2454] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014bc50d0c5a44bdbf49c0eb41fa93ad in term 0.
I20240501 07:59:47.085130 3614 leader_election.cc:302] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 40c9ccd5f29c424eb845d670a6f7d21e; no voters:
I20240501 07:59:47.085309 4564 raft_consensus.cc:2781] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:47.085371 4564 raft_consensus.cc:482] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:47.085399 4564 raft_consensus.cc:3037] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.086254 4564 raft_consensus.cc:504] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:47.086550 4564 leader_election.cc:288] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 1 election: Requested vote from peers 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:47.086948 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0cda2d32534bb29a6a2538ac13368a" candidate_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
I20240501 07:59:47.087065 3809 raft_consensus.cc:3037] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.087922 3809 raft_consensus.cc:2454] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014bc50d0c5a44bdbf49c0eb41fa93ad in term 1.
I20240501 07:59:47.088383 3614 leader_election.cc:302] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 40c9ccd5f29c424eb845d670a6f7d21e; no voters:
I20240501 07:59:47.088565 4564 raft_consensus.cc:2781] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:47.088737 4564 raft_consensus.cc:686] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Becoming Leader. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Running, Role: LEADER
I20240501 07:59:47.088913 4564 consensus_queue.cc:237] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:47.090617 3532 catalog_manager.cc:5484] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129). New cstate: current_term: 1 leader_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:47.235957 4557 raft_consensus.cc:482] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:47.236097 4557 raft_consensus.cc:504] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:47.236514 4557 leader_election.cc:288] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:47.237150 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc31e13bb07f44ddb7b80282ee4c434b" candidate_uuid: "8635309522a045b1941fea81699d524c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" is_pre_election: true
I20240501 07:59:47.237293 3678 raft_consensus.cc:2454] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8635309522a045b1941fea81699d524c in term 0.
I20240501 07:59:47.237610 3876 leader_election.cc:302] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:47.237783 4557 raft_consensus.cc:2781] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:47.237844 4557 raft_consensus.cc:482] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:47.237905 4557 raft_consensus.cc:3037] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.238734 4557 raft_consensus.cc:504] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:47.239015 4557 leader_election.cc:288] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 election: Requested vote from peers 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:47.239409 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc31e13bb07f44ddb7b80282ee4c434b" candidate_uuid: "8635309522a045b1941fea81699d524c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
I20240501 07:59:47.239532 3678 raft_consensus.cc:3037] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.240329 3678 raft_consensus.cc:2454] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8635309522a045b1941fea81699d524c in term 1.
I20240501 07:59:47.240636 3876 leader_election.cc:302] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:47.240808 4557 raft_consensus.cc:2781] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:47.240906 4557 raft_consensus.cc:686] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Becoming Leader. State: Replica: 8635309522a045b1941fea81699d524c, State: Running, Role: LEADER
I20240501 07:59:47.241055 4557 consensus_queue.cc:237] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:47.242578 3532 catalog_manager.cc:5484] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8635309522a045b1941fea81699d524c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:47.432649 4671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:47.432917 4671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:47.436268 4671 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 07:59:47.477347 4623 consensus_queue.cc:1035] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:47.518255 3809 raft_consensus.cc:1261] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:47.518679 4564 consensus_queue.cc:1035] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:47.519418 3678 raft_consensus.cc:1261] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:47.519862 4557 consensus_queue.cc:1035] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 84.3913 ms
time per row: 0.0421956 ms
I20240501 07:59:47.566157 3532 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55978:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:47.571930 3659 tablet_service.cc:1467] Processing CreateTablet for tablet b23c9dc72fbc4e409f1d22506b11cb42 (DEFAULT_TABLE table=rebalance_test_table_3 [id=86dc286087384ca29a85396def16cf86]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.571880 3789 tablet_service.cc:1467] Processing CreateTablet for tablet b23c9dc72fbc4e409f1d22506b11cb42 (DEFAULT_TABLE table=rebalance_test_table_3 [id=86dc286087384ca29a85396def16cf86]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.572238 3659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23c9dc72fbc4e409f1d22506b11cb42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.572238 3789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23c9dc72fbc4e409f1d22506b11cb42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.572624 3921 tablet_service.cc:1467] Processing CreateTablet for tablet c02a65bc31814809ae316ae5fdd07a6e (DEFAULT_TABLE table=rebalance_test_table_3 [id=86dc286087384ca29a85396def16cf86]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.572913 3921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02a65bc31814809ae316ae5fdd07a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.574602 4730 tablet_bootstrap.cc:492] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Bootstrap starting.
I20240501 07:59:47.574888 3920 tablet_service.cc:1467] Processing CreateTablet for tablet 839df5100c814d6bb78dc67a8c834a49 (DEFAULT_TABLE table=rebalance_test_table_3 [id=86dc286087384ca29a85396def16cf86]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.574956 4731 tablet_bootstrap.cc:492] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c: Bootstrap starting.
I20240501 07:59:47.575011 4729 tablet_bootstrap.cc:492] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e: Bootstrap starting.
I20240501 07:59:47.575122 3920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 839df5100c814d6bb78dc67a8c834a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.575570 4730 tablet_bootstrap.cc:654] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.575786 4729 tablet_bootstrap.cc:654] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.575786 4731 tablet_bootstrap.cc:654] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.577139 4731 tablet_bootstrap.cc:492] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c: No bootstrap required, opened a new log
I20240501 07:59:47.577231 4729 tablet_bootstrap.cc:492] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e: No bootstrap required, opened a new log
I20240501 07:59:47.577289 4731 ts_tablet_manager.cc:1367] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:47.577327 4729 ts_tablet_manager.cc:1367] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:47.577378 3790 tablet_service.cc:1467] Processing CreateTablet for tablet c02a65bc31814809ae316ae5fdd07a6e (DEFAULT_TABLE table=rebalance_test_table_3 [id=86dc286087384ca29a85396def16cf86]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.577620 3790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02a65bc31814809ae316ae5fdd07a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.577673 3658 tablet_service.cc:1467] Processing CreateTablet for tablet 839df5100c814d6bb78dc67a8c834a49 (DEFAULT_TABLE table=rebalance_test_table_3 [id=86dc286087384ca29a85396def16cf86]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.577766 4731 raft_consensus.cc:348] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:47.577869 4731 raft_consensus.cc:374] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.577809 4729 raft_consensus.cc:348] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:47.577908 4731 raft_consensus.cc:729] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8635309522a045b1941fea81699d524c, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.577917 4729 raft_consensus.cc:374] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.577952 4729 raft_consensus.cc:729] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.577968 3658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 839df5100c814d6bb78dc67a8c834a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.578090 4731 consensus_queue.cc:260] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:47.578117 4729 consensus_queue.cc:260] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:47.578414 4729 ts_tablet_manager.cc:1398] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:47.578588 4731 ts_tablet_manager.cc:1398] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:47.578738 4731 tablet_bootstrap.cc:492] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c: Bootstrap starting.
I20240501 07:59:47.579648 4731 tablet_bootstrap.cc:654] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.579972 4729 tablet_bootstrap.cc:492] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e: Bootstrap starting.
I20240501 07:59:47.580806 4731 tablet_bootstrap.cc:492] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c: No bootstrap required, opened a new log
I20240501 07:59:47.580888 4729 tablet_bootstrap.cc:654] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.580947 4731 ts_tablet_manager.cc:1367] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:47.581539 4731 raft_consensus.cc:348] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:47.581708 4731 raft_consensus.cc:374] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.581799 4731 raft_consensus.cc:729] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8635309522a045b1941fea81699d524c, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.582013 4731 consensus_queue.cc:260] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:47.582088 4729 tablet_bootstrap.cc:492] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e: No bootstrap required, opened a new log
I20240501 07:59:47.582161 4729 ts_tablet_manager.cc:1367] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:47.582175 4730 tablet_bootstrap.cc:492] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad: No bootstrap required, opened a new log
I20240501 07:59:47.582265 4730 ts_tablet_manager.cc:1367] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:47.582389 4731 ts_tablet_manager.cc:1398] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:47.582599 4729 raft_consensus.cc:348] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:47.582687 4729 raft_consensus.cc:374] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.582726 4729 raft_consensus.cc:729] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.582732 4730 raft_consensus.cc:348] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:47.582834 4730 raft_consensus.cc:374] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.582866 4730 raft_consensus.cc:729] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.582854 4729 consensus_queue.cc:260] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:47.583009 4730 consensus_queue.cc:260] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:47.583117 4729 ts_tablet_manager.cc:1398] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:47.583268 4730 ts_tablet_manager.cc:1398] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:47.583402 4730 tablet_bootstrap.cc:492] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Bootstrap starting.
I20240501 07:59:47.584327 4730 tablet_bootstrap.cc:654] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.585436 4730 tablet_bootstrap.cc:492] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad: No bootstrap required, opened a new log
I20240501 07:59:47.585530 4730 ts_tablet_manager.cc:1367] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:47.585911 4730 raft_consensus.cc:348] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:47.586064 4730 raft_consensus.cc:374] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.586108 4730 raft_consensus.cc:729] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.586226 4730 consensus_queue.cc:260] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:47.586512 4730 ts_tablet_manager.cc:1398] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:47.659796 4557 raft_consensus.cc:482] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:47.659945 4557 raft_consensus.cc:504] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:47.660313 4557 leader_election.cc:288] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:47.660859 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02a65bc31814809ae316ae5fdd07a6e" candidate_uuid: "8635309522a045b1941fea81699d524c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" is_pre_election: true
I20240501 07:59:47.661002 3809 raft_consensus.cc:2454] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8635309522a045b1941fea81699d524c in term 0.
I20240501 07:59:47.661288 3876 leader_election.cc:302] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 40c9ccd5f29c424eb845d670a6f7d21e, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:47.661453 4557 raft_consensus.cc:2781] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:47.661512 4557 raft_consensus.cc:482] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:47.661562 4557 raft_consensus.cc:3037] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.662454 4557 raft_consensus.cc:504] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:47.662745 4557 leader_election.cc:288] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 election: Requested vote from peers 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:47.663126 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02a65bc31814809ae316ae5fdd07a6e" candidate_uuid: "8635309522a045b1941fea81699d524c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
I20240501 07:59:47.663256 3809 raft_consensus.cc:3037] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.663975 3809 raft_consensus.cc:2454] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8635309522a045b1941fea81699d524c in term 1.
I20240501 07:59:47.664299 3876 leader_election.cc:302] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 40c9ccd5f29c424eb845d670a6f7d21e, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:47.664459 4557 raft_consensus.cc:2781] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:47.664561 4557 raft_consensus.cc:686] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Becoming Leader. State: Replica: 8635309522a045b1941fea81699d524c, State: Running, Role: LEADER
I20240501 07:59:47.664693 4557 consensus_queue.cc:237] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:47.666456 3532 catalog_manager.cc:5484] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8635309522a045b1941fea81699d524c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:47.669734 4557 raft_consensus.cc:482] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:47.669837 4557 raft_consensus.cc:504] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:47.670162 4557 leader_election.cc:288] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:47.670603 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "839df5100c814d6bb78dc67a8c834a49" candidate_uuid: "8635309522a045b1941fea81699d524c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" is_pre_election: true
I20240501 07:59:47.670746 3678 raft_consensus.cc:2454] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8635309522a045b1941fea81699d524c in term 0.
I20240501 07:59:47.671047 3876 leader_election.cc:302] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:47.671202 4557 raft_consensus.cc:2781] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:47.671267 4557 raft_consensus.cc:482] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:47.671303 4557 raft_consensus.cc:3037] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.672011 4557 raft_consensus.cc:504] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:47.672248 4557 leader_election.cc:288] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 election: Requested vote from peers 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:47.672619 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "839df5100c814d6bb78dc67a8c834a49" candidate_uuid: "8635309522a045b1941fea81699d524c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
I20240501 07:59:47.672744 3678 raft_consensus.cc:3037] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.673529 3678 raft_consensus.cc:2454] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8635309522a045b1941fea81699d524c in term 1.
I20240501 07:59:47.673815 3876 leader_election.cc:302] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:47.674019 4557 raft_consensus.cc:2781] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:47.674115 4557 raft_consensus.cc:686] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Becoming Leader. State: Replica: 8635309522a045b1941fea81699d524c, State: Running, Role: LEADER
I20240501 07:59:47.674257 4557 consensus_queue.cc:237] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } }
I20240501 07:59:47.675927 3532 catalog_manager.cc:5484] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8635309522a045b1941fea81699d524c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:47.805495 4725 raft_consensus.cc:482] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:47.805647 4725 raft_consensus.cc:504] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:47.806151 4725 leader_election.cc:288] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:47.806622 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23c9dc72fbc4e409f1d22506b11cb42" candidate_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" is_pre_election: true
I20240501 07:59:47.806797 3809 raft_consensus.cc:2454] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014bc50d0c5a44bdbf49c0eb41fa93ad in term 0.
I20240501 07:59:47.807060 3614 leader_election.cc:302] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 40c9ccd5f29c424eb845d670a6f7d21e; no voters:
I20240501 07:59:47.807271 4725 raft_consensus.cc:2781] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:47.807322 4725 raft_consensus.cc:482] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:47.807363 4725 raft_consensus.cc:3037] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.808316 4725 raft_consensus.cc:504] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:47.808585 4725 leader_election.cc:288] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 1 election: Requested vote from peers 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:47.808938 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23c9dc72fbc4e409f1d22506b11cb42" candidate_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
I20240501 07:59:47.809072 3809 raft_consensus.cc:3037] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.809844 3809 raft_consensus.cc:2454] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014bc50d0c5a44bdbf49c0eb41fa93ad in term 1.
I20240501 07:59:47.810161 3614 leader_election.cc:302] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 40c9ccd5f29c424eb845d670a6f7d21e; no voters:
I20240501 07:59:47.810333 4725 raft_consensus.cc:2781] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:47.810422 4725 raft_consensus.cc:686] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Becoming Leader. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Running, Role: LEADER
I20240501 07:59:47.810562 4725 consensus_queue.cc:237] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:47.812383 3532 catalog_manager.cc:5484] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129). New cstate: current_term: 1 leader_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: HEALTHY } } }
W20240501 07:59:47.928875 4741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:47.929162 4741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:47.932283 4741 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 07:59:48.011031 3678 raft_consensus.cc:1261] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.011557 4557 consensus_queue.cc:1035] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:48.013660 3809 raft_consensus.cc:1261] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.014843 3810 raft_consensus.cc:1261] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.015376 4725 consensus_queue.cc:1035] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:48.016525 4557 consensus_queue.cc:1035] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 83.7407 ms
time per row: 0.0418704 ms
I20240501 07:59:48.060395 3531 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55978:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:48.066035 3790 tablet_service.cc:1467] Processing CreateTablet for tablet 393b4eef45ff435abd0fde4aec22da88 (DEFAULT_TABLE table=rebalance_test_table_4 [id=980322bb0877411f904d1f982f67275b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.066066 3920 tablet_service.cc:1467] Processing CreateTablet for tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b (DEFAULT_TABLE table=rebalance_test_table_4 [id=980322bb0877411f904d1f982f67275b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.066056 3658 tablet_service.cc:1467] Processing CreateTablet for tablet 393b4eef45ff435abd0fde4aec22da88 (DEFAULT_TABLE table=rebalance_test_table_4 [id=980322bb0877411f904d1f982f67275b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.066386 3920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.066385 3658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393b4eef45ff435abd0fde4aec22da88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.066388 3790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393b4eef45ff435abd0fde4aec22da88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.068161 4729 tablet_bootstrap.cc:492] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e: Bootstrap starting.
I20240501 07:59:48.068536 4730 tablet_bootstrap.cc:492] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Bootstrap starting.
I20240501 07:59:48.068686 4731 tablet_bootstrap.cc:492] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c: Bootstrap starting.
I20240501 07:59:48.069123 4729 tablet_bootstrap.cc:654] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.069334 4730 tablet_bootstrap.cc:654] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.069453 4731 tablet_bootstrap.cc:654] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.069478 3790 tablet_service.cc:1467] Processing CreateTablet for tablet 54a2b2f834494f29b306e80e48804a43 (DEFAULT_TABLE table=rebalance_test_table_4 [id=980322bb0877411f904d1f982f67275b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.069687 3790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a2b2f834494f29b306e80e48804a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.070377 3920 tablet_service.cc:1467] Processing CreateTablet for tablet 54a2b2f834494f29b306e80e48804a43 (DEFAULT_TABLE table=rebalance_test_table_4 [id=980322bb0877411f904d1f982f67275b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.070622 3920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a2b2f834494f29b306e80e48804a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.070720 4730 tablet_bootstrap.cc:492] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad: No bootstrap required, opened a new log
I20240501 07:59:48.070791 4730 ts_tablet_manager.cc:1367] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:48.071031 4729 tablet_bootstrap.cc:492] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e: No bootstrap required, opened a new log
I20240501 07:59:48.071101 4729 ts_tablet_manager.cc:1367] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:48.071252 4730 raft_consensus.cc:348] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:48.071360 4730 raft_consensus.cc:374] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.071393 4730 raft_consensus.cc:729] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.071522 4730 consensus_queue.cc:260] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:48.071573 4729 raft_consensus.cc:348] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:48.071666 4729 raft_consensus.cc:374] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.071696 4729 raft_consensus.cc:729] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.071810 4730 ts_tablet_manager.cc:1398] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:48.071800 4729 consensus_queue.cc:260] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:48.072063 4729 ts_tablet_manager.cc:1398] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:48.072088 3658 tablet_service.cc:1467] Processing CreateTablet for tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b (DEFAULT_TABLE table=rebalance_test_table_4 [id=980322bb0877411f904d1f982f67275b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.072227 4729 tablet_bootstrap.cc:492] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e: Bootstrap starting.
I20240501 07:59:48.072319 3658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.072609 4731 tablet_bootstrap.cc:492] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c: No bootstrap required, opened a new log
I20240501 07:59:48.072682 4731 ts_tablet_manager.cc:1367] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:59:48.073102 4729 tablet_bootstrap.cc:654] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.073140 4731 raft_consensus.cc:348] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:48.073237 4731 raft_consensus.cc:374] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.073276 4731 raft_consensus.cc:729] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8635309522a045b1941fea81699d524c, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.073392 4731 consensus_queue.cc:260] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:48.073644 4731 ts_tablet_manager.cc:1398] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:59:48.073801 4731 tablet_bootstrap.cc:492] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c: Bootstrap starting.
I20240501 07:59:48.074972 4731 tablet_bootstrap.cc:654] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.075152 4730 tablet_bootstrap.cc:492] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad: Bootstrap starting.
I20240501 07:59:48.076072 4730 tablet_bootstrap.cc:654] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.077049 4729 tablet_bootstrap.cc:492] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e: No bootstrap required, opened a new log
I20240501 07:59:48.077152 4729 ts_tablet_manager.cc:1367] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:48.077325 4731 tablet_bootstrap.cc:492] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c: No bootstrap required, opened a new log
I20240501 07:59:48.077405 4730 tablet_bootstrap.cc:492] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad: No bootstrap required, opened a new log
I20240501 07:59:48.077426 4731 ts_tablet_manager.cc:1367] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:48.077499 4730 ts_tablet_manager.cc:1367] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:48.077764 4729 raft_consensus.cc:348] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:48.077812 4731 raft_consensus.cc:348] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:48.077891 4731 raft_consensus.cc:374] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.077891 4729 raft_consensus.cc:374] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.077925 4729 raft_consensus.cc:729] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.077925 4731 raft_consensus.cc:729] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8635309522a045b1941fea81699d524c, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.077872 4730 raft_consensus.cc:348] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:48.078018 4730 raft_consensus.cc:374] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.078065 4730 raft_consensus.cc:729] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.078086 4731 consensus_queue.cc:260] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:48.078145 4729 consensus_queue.cc:260] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:48.078190 4730 consensus_queue.cc:260] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:48.078362 4731 ts_tablet_manager.cc:1398] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:48.078406 4729 ts_tablet_manager.cc:1398] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:48.078464 4730 ts_tablet_manager.cc:1398] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:48.173457 4623 raft_consensus.cc:482] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:48.173602 4623 raft_consensus.cc:504] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:48.174041 4623 leader_election.cc:288] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:48.174597 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393b4eef45ff435abd0fde4aec22da88" candidate_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" is_pre_election: true
I20240501 07:59:48.174765 3678 raft_consensus.cc:2454] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c9ccd5f29c424eb845d670a6f7d21e in term 0.
I20240501 07:59:48.175114 3745 leader_election.cc:302] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 40c9ccd5f29c424eb845d670a6f7d21e; no voters:
I20240501 07:59:48.175298 4623 raft_consensus.cc:2781] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:48.175366 4623 raft_consensus.cc:482] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:48.175406 4623 raft_consensus.cc:3037] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.176218 4623 raft_consensus.cc:504] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:48.176487 4623 leader_election.cc:288] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 1 election: Requested vote from peers 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:48.176891 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393b4eef45ff435abd0fde4aec22da88" candidate_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
I20240501 07:59:48.177008 3678 raft_consensus.cc:3037] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.177791 3678 raft_consensus.cc:2454] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c9ccd5f29c424eb845d670a6f7d21e in term 1.
I20240501 07:59:48.178113 3745 leader_election.cc:302] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 40c9ccd5f29c424eb845d670a6f7d21e; no voters:
I20240501 07:59:48.178287 4623 raft_consensus.cc:2781] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:48.178376 4623 raft_consensus.cc:686] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Becoming Leader. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Running, Role: LEADER
I20240501 07:59:48.178541 4623 consensus_queue.cc:237] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:48.180271 3529 catalog_manager.cc:5484] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130). New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:48.279546 4770 raft_consensus.cc:482] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:48.279701 4770 raft_consensus.cc:504] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:48.280086 4770 leader_election.cc:288] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:48.280648 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a0bc2519cd43eda2f2fc8b5a6e7d5b" candidate_uuid: "8635309522a045b1941fea81699d524c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" is_pre_election: true
I20240501 07:59:48.280807 3678 raft_consensus.cc:2454] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8635309522a045b1941fea81699d524c in term 0.
I20240501 07:59:48.281152 3876 leader_election.cc:302] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:48.281356 4770 raft_consensus.cc:2781] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:48.281435 4770 raft_consensus.cc:482] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:48.281483 4770 raft_consensus.cc:3037] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.282440 4770 raft_consensus.cc:504] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:48.282747 4770 leader_election.cc:288] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 election: Requested vote from peers 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:48.283241 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a0bc2519cd43eda2f2fc8b5a6e7d5b" candidate_uuid: "8635309522a045b1941fea81699d524c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
I20240501 07:59:48.283354 3678 raft_consensus.cc:3037] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.284132 3678 raft_consensus.cc:2454] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8635309522a045b1941fea81699d524c in term 1.
I20240501 07:59:48.284462 3876 leader_election.cc:302] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:48.284679 4770 raft_consensus.cc:2781] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:48.284770 4770 raft_consensus.cc:686] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Becoming Leader. State: Replica: 8635309522a045b1941fea81699d524c, State: Running, Role: LEADER
I20240501 07:59:48.284914 4770 consensus_queue.cc:237] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } }
I20240501 07:59:48.286752 3529 catalog_manager.cc:5484] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8635309522a045b1941fea81699d524c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:48.293002 4770 raft_consensus.cc:482] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:48.293128 4770 raft_consensus.cc:504] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:48.293475 4770 leader_election.cc:288] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:48.294165 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a2b2f834494f29b306e80e48804a43" candidate_uuid: "8635309522a045b1941fea81699d524c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" is_pre_election: true
I20240501 07:59:48.294319 3809 raft_consensus.cc:2454] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8635309522a045b1941fea81699d524c in term 0.
I20240501 07:59:48.294672 3876 leader_election.cc:302] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 40c9ccd5f29c424eb845d670a6f7d21e, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:48.294853 4770 raft_consensus.cc:2781] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:48.294914 4770 raft_consensus.cc:482] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:48.295014 4770 raft_consensus.cc:3037] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.295830 4770 raft_consensus.cc:504] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:48.296267 4770 leader_election.cc:288] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 election: Requested vote from peers 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:48.296921 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a2b2f834494f29b306e80e48804a43" candidate_uuid: "8635309522a045b1941fea81699d524c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
I20240501 07:59:48.297192 3809 raft_consensus.cc:3037] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.298287 3809 raft_consensus.cc:2454] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8635309522a045b1941fea81699d524c in term 1.
I20240501 07:59:48.298660 3876 leader_election.cc:302] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 40c9ccd5f29c424eb845d670a6f7d21e, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:48.298833 4770 raft_consensus.cc:2781] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:48.298974 4770 raft_consensus.cc:686] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Becoming Leader. State: Replica: 8635309522a045b1941fea81699d524c, State: Running, Role: LEADER
I20240501 07:59:48.299168 4770 consensus_queue.cc:237] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } }
I20240501 07:59:48.300833 3529 catalog_manager.cc:5484] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8635309522a045b1941fea81699d524c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:48.417201 4780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:48.417480 4780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:48.420696 4780 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 07:59:48.511456 3809 raft_consensus.cc:1261] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.511898 4770 consensus_queue.cc:1035] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:48.512193 3678 raft_consensus.cc:1261] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.512631 4770 consensus_queue.cc:1035] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:48.514026 3679 raft_consensus.cc:1261] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.515050 4623 consensus_queue.cc:1035] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 101.8 ms
time per row: 0.0508999 ms
I20240501 07:59:48.566485 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:37117
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:44911
--webserver_port=40577
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 07:59:48.678575 4810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:48.678843 4810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:48.678887 4810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:48.682123 4810 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:48.682200 4810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:48.682271 4810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:59:48.685926 4810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:37117
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=40577
--tserver_master_addrs=127.0.99.190:44911
--never_fsync=true
--heap_profile_path=/tmp/kudu.4810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:48.686838 4810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:48.688023 4810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:48.697876 4815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:48.698674 4810 server_base.cc:1034] running on GCE node
W20240501 07:59:48.699321 4818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:48.700551 4816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:48.701020 4810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:48.701663 4810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:48.702827 4810 hybrid_clock.cc:648] HybridClock initialized: now 1714550388702817 us; error 39 us; skew 500 ppm
I20240501 07:59:48.704722 4810 webserver.cc:460] Webserver started at http://127.0.99.132:40577/ using document root <none> and password file <none>
I20240501 07:59:48.705322 4810 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:48.705386 4810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:48.708796 4810 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:48.710739 4824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:48.711738 4810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:48.711834 4810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d8461c1ff1ed48b09047faeea87e9598"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:48.712289 4810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:48.726661 4810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:48.727329 4810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:48.727491 4810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:48.728037 4810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:48.729152 4831 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 07:59:48.732272 4810 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 07:59:48.732329 4810 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240501 07:59:48.732380 4810 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 07:59:48.733835 4810 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 07:59:48.733878 4810 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240501 07:59:48.734236 4831 tablet_bootstrap.cc:492] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598: Bootstrap starting.
I20240501 07:59:48.744200 4810 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:37117
I20240501 07:59:48.744266 4938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:37117 every 8 connection(s)
I20240501 07:59:48.745415 4810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:59:48.749167 4831 log.cc:824] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:48.750473 4831 tablet_bootstrap.cc:492] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:48.750753 4831 tablet_bootstrap.cc:492] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598: Bootstrap complete.
I20240501 07:59:48.751816 4831 ts_tablet_manager.cc:1367] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20240501 07:59:48.752485 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 4810
I20240501 07:59:48.752836 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:40225
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:44911
--webserver_port=45189
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:48.754765 4831 raft_consensus.cc:348] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } }
I20240501 07:59:48.755100 4831 raft_consensus.cc:729] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8461c1ff1ed48b09047faeea87e9598, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.755590 4831 consensus_queue.cc:260] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } }
I20240501 07:59:48.755887 4939 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44911
I20240501 07:59:48.756106 4939 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:48.756563 4831 ts_tablet_manager.cc:1398] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 07:59:48.756646 4939 heartbeater.cc:511] Master 127.0.99.190:44911 requested a full tablet report, sending...
I20240501 07:59:48.758560 3529 ts_manager.cc:194] Re-registered known tserver with Master: d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132:37117)
I20240501 07:59:48.759335 3529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:57301
I20240501 07:59:48.761085 4939 heartbeater.cc:503] Master 127.0.99.190:44911 was elected leader, sending a full tablet report...
W20240501 07:59:48.865928 4943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:48.866190 4943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:48.866259 4943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:48.869266 4943 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:48.869342 4943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:48.869447 4943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:59:48.873075 4943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:40225
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=45189
--tserver_master_addrs=127.0.99.190:44911
--never_fsync=true
--heap_profile_path=/tmp/kudu.4943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:48.874082 4943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:48.875238 4943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:48.884236 4952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:48.885182 4950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:48.886049 4949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:48.886893 4943 server_base.cc:1034] running on GCE node
I20240501 07:59:48.887313 4943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:48.887914 4943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:48.889062 4943 hybrid_clock.cc:648] HybridClock initialized: now 1714550388889050 us; error 29 us; skew 500 ppm
I20240501 07:59:48.891132 4943 webserver.cc:460] Webserver started at http://127.0.99.133:45189/ using document root <none> and password file <none>
I20240501 07:59:48.891690 4943 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:48.891759 4943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:48.895352 4943 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:48.897315 4958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:48.898381 4943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:48.898494 4943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "da103aa972ab49889fc0bce4abc56bf7"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:48.898979 4943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:48.932399 4943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:48.933128 4943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:48.933311 4943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:48.933884 4943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:48.934871 4943 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:48.934923 4943 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:48.934988 4943 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:48.935020 4943 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:48.943910 4943 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:40225
I20240501 07:59:48.944160 5071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:40225 every 8 connection(s)
I20240501 07:59:48.945283 4943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:59:48.949784 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 4943
I20240501 07:59:48.950246 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:36011
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:44911
--webserver_port=46473
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:48.955762 5072 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44911
I20240501 07:59:48.956043 5072 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:48.956826 5072 heartbeater.cc:511] Master 127.0.99.190:44911 requested a full tablet report, sending...
I20240501 07:59:48.958247 3529 ts_manager.cc:194] Re-registered known tserver with Master: da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133:40225)
I20240501 07:59:48.959089 3529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:59967
W20240501 07:59:48.996498 4940 tablet.cc:2369] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:59:49.066711 5075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:49.067001 5075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:49.067076 5075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:49.070233 5075 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:49.070312 5075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:49.070405 5075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:59:49.074599 5075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:36011
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=46473
--tserver_master_addrs=127.0.99.190:44911
--never_fsync=true
--heap_profile_path=/tmp/kudu.5075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:49.075563 5075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:49.076745 5075 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:49.085783 5083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:49.086545 5080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:49.087323 5081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:49.088536 5075 server_base.cc:1034] running on GCE node
I20240501 07:59:49.089027 5075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:49.089654 5075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:49.090816 5075 hybrid_clock.cc:648] HybridClock initialized: now 1714550389090795 us; error 35 us; skew 500 ppm
I20240501 07:59:49.092845 5075 webserver.cc:460] Webserver started at http://127.0.99.134:46473/ using document root <none> and password file <none>
I20240501 07:59:49.093492 5075 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:49.093582 5075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:49.097193 5075 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:49.099298 5089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:49.100363 5075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:49.100486 5075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6208df88dcc34b4daf3d27a574e53a95"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-wslg"
I20240501 07:59:49.100942 5075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:49.121955 5075 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:49.122697 5075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:49.122846 5075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:49.123363 5075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:49.124230 5075 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:49.124286 5075 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:49.124327 5075 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:49.124348 5075 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:49.134212 5075 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:36011
I20240501 07:59:49.134250 5202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:36011 every 8 connection(s)
I20240501 07:59:49.135533 5075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:59:49.138156 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 5075
I20240501 07:59:49.138589 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:41559
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:44911
--webserver_port=41587
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:49.145450 5203 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44911
I20240501 07:59:49.145707 5203 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:49.146320 5203 heartbeater.cc:511] Master 127.0.99.190:44911 requested a full tablet report, sending...
I20240501 07:59:49.147262 3529 ts_manager.cc:194] Re-registered known tserver with Master: 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134:36011)
I20240501 07:59:49.147821 3529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:34031
W20240501 07:59:49.255373 5206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:49.255631 5206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:49.255677 5206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:49.258838 5206 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:49.258913 5206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:49.259018 5206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:59:49.262822 5206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:41559
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=41587
--tserver_master_addrs=127.0.99.190:44911
--never_fsync=true
--heap_profile_path=/tmp/kudu.5206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:49.263796 5206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:49.264981 5206 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:49.274318 5212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:49.275295 5211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:49.276167 5214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:49.277209 5206 server_base.cc:1034] running on GCE node
I20240501 07:59:49.277745 5206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:49.278488 5206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:49.279665 5206 hybrid_clock.cc:648] HybridClock initialized: now 1714550389279626 us; error 58 us; skew 500 ppm
I20240501 07:59:49.282028 5206 webserver.cc:460] Webserver started at http://127.0.99.135:41587/ using document root <none> and password file <none>
I20240501 07:59:49.282620 5206 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:49.282703 5206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:49.286540 5206 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20240501 07:59:49.288570 5220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:49.289711 5206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:49.289814 5206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "09d8fa5c011b4e398612ba0274c14f2e"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-wslg"
I20240501 07:59:49.290345 5206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:49.319423 5206 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:49.320111 5206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:49.320263 5206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:49.320761 5206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:49.321838 5227 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 07:59:49.325044 5206 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 07:59:49.325129 5206 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240501 07:59:49.325192 5206 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 07:59:49.326779 5206 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 07:59:49.326824 5206 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:49.327064 5227 tablet_bootstrap.cc:492] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e: Bootstrap starting.
I20240501 07:59:49.337738 5206 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:41559
I20240501 07:59:49.338057 5334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:41559 every 8 connection(s)
I20240501 07:59:49.338955 5206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:59:49.342625 5227 log.cc:824] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:49.345172 5227 tablet_bootstrap.cc:492] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:59:49.345638 5227 tablet_bootstrap.cc:492] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e: Bootstrap complete.
I20240501 07:59:49.346990 5227 ts_tablet_manager.cc:1367] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.010s
I20240501 07:59:49.348043 5335 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44911
I20240501 07:59:49.348264 5335 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:49.348387 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 5206
I20240501 07:59:49.348575 398 test_util.cc:268] Using random seed: -555214625
I20240501 07:59:49.348904 5335 heartbeater.cc:511] Master 127.0.99.190:44911 requested a full tablet report, sending...
I20240501 07:59:49.350538 3529 ts_manager.cc:194] Re-registered known tserver with Master: 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135:41559)
I20240501 07:59:49.350404 5227 raft_consensus.cc:348] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } }
I20240501 07:59:49.350970 5227 raft_consensus.cc:729] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d8fa5c011b4e398612ba0274c14f2e, State: Initialized, Role: FOLLOWER
I20240501 07:59:49.351369 3529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:34663
I20240501 07:59:49.351449 5227 consensus_queue.cc:260] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } }
I20240501 07:59:49.352294 5227 ts_tablet_manager.cc:1398] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20240501 07:59:49.353283 5335 heartbeater.cc:503] Master 127.0.99.190:44911 was elected leader, sending a full tablet report...
I20240501 07:59:49.360756 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240501 07:59:49.360966 398 test_util.cc:268] Using random seed: -555202234
I20240501 07:59:49.369917 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240501 07:59:49.370121 398 test_util.cc:268] Using random seed: -555193081
I20240501 07:59:49.381744 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240501 07:59:49.381932 398 test_util.cc:268] Using random seed: -555181271
I20240501 07:59:49.397336 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240501 07:59:49.397521 398 test_util.cc:268] Using random seed: -555165682
I20240501 07:59:49.425382 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240501 07:59:49.590083 5336 tablet.cc:2369] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:49.860970 5006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:49.867584 3790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:49.871446 5137 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:49.875689 3920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:49.889859 4871 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:49.908775 3658 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:49.915450 5268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:49.961759 5072 heartbeater.cc:503] Master 127.0.99.190:44911 was elected leader, sending a full tablet report...
I20240501 07:59:50.149905 5203 heartbeater.cc:503] Master 127.0.99.190:44911 was elected leader, sending a full tablet report...
I20240501 07:59:50.194185 5006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:50.223950 3790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:50.231050 5137 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:50.236748 3658 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:50.239254 5268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:50.244040 3920 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:50.246630 4871 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:50.335984 5519 raft_consensus.cc:482] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:50.336362 5519 raft_consensus.cc:504] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } }
I20240501 07:59:50.337606 5519 leader_election.cc:288] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135:41559)
I20240501 07:59:50.353446 3806 consensus_queue.cc:237] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.35, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } }
I20240501 07:59:50.355046 5279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a2915d4efd4d48a1d41df5ecf2f4d7" candidate_uuid: "d8461c1ff1ed48b09047faeea87e9598" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09d8fa5c011b4e398612ba0274c14f2e" is_pre_election: true
I20240501 07:59:50.355325 5279 raft_consensus.cc:2396] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8461c1ff1ed48b09047faeea87e9598 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240501 07:59:50.355473 3939 raft_consensus.cc:1261] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20240501 07:59:50.355841 4623 consensus_queue.cc:1035] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240501 07:59:50.356060 4826 leader_election.cc:302] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: d8461c1ff1ed48b09047faeea87e9598; no voters: 09d8fa5c011b4e398612ba0274c14f2e
I20240501 07:59:50.356585 5519 raft_consensus.cc:2726] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:50.357949 5409 raft_consensus.cc:2932] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Committing config change with OpId 1.36: config changed from index -1 to 36, NON_VOTER 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } } }
I20240501 07:59:50.360220 3939 raft_consensus.cc:2932] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index -1 to 36, NON_VOTER 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } } }
I20240501 07:59:50.360792 3531 catalog_manager.cc:5484] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e reported cstate change: config changed from index -1 to 36, NON_VOTER 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:50.371536 3746 consensus_peers.cc:485] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e -> Peer 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134:36011): Couldn't send request to peer 6208df88dcc34b4daf3d27a574e53a95. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed30e32a4ba2483484b5bc5e3cfb44fd. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:50.382465 3806 consensus_queue.cc:237] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 1.41, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } }
I20240501 07:59:50.383904 3677 raft_consensus.cc:1261] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 41. Preceding OpId from leader: term: 1 index: 42. (index mismatch)
I20240501 07:59:50.384811 4623 consensus_queue.cc:1035] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20240501 07:59:50.389693 4623 raft_consensus.cc:2932] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Committing config change with OpId 1.42: config changed from index -1 to 42, NON_VOTER da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) added. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } } }
I20240501 07:59:50.390259 3677 raft_consensus.cc:2932] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Committing config change with OpId 1.42: config changed from index -1 to 42, NON_VOTER da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) added. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } } }
W20240501 07:59:50.392055 3743 consensus_peers.cc:485] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e -> Peer da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133:40225): Couldn't send request to peer da103aa972ab49889fc0bce4abc56bf7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 393b4eef45ff435abd0fde4aec22da88. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:50.394488 3528 catalog_manager.cc:5484] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: config changed from index -1 to 42, NON_VOTER da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } } }
I20240501 07:59:50.446058 3808 consensus_queue.cc:237] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.47, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } }
I20240501 07:59:50.447674 3939 raft_consensus.cc:1261] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 1 index: 48. (index mismatch)
I20240501 07:59:50.448220 5371 consensus_queue.cc:1035] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20240501 07:59:50.453541 5410 raft_consensus.cc:2932] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Committing config change with OpId 1.48: config changed from index -1 to 48, NON_VOTER d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } } }
I20240501 07:59:50.454327 3941 raft_consensus.cc:2932] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Committing config change with OpId 1.48: config changed from index -1 to 48, NON_VOTER d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } } }
W20240501 07:59:50.454968 3745 consensus_peers.cc:485] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e -> Peer d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132:37117): Couldn't send request to peer d8461c1ff1ed48b09047faeea87e9598. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f75acaf04c12462aa4b3fa7e019776d4. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:50.456285 3531 catalog_manager.cc:5484] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e reported cstate change: config changed from index -1 to 48, NON_VOTER d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:50.476315 3939 consensus_queue.cc:237] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } }
I20240501 07:59:50.478811 3677 raft_consensus.cc:1261] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 07:59:50.480803 5431 consensus_queue.cc:1035] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 07:59:50.486450 5414 raft_consensus.cc:2932] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } } }
W20240501 07:59:50.489092 3875 consensus_peers.cc:485] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c -> Peer 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135:41559): Couldn't send request to peer 09d8fa5c011b4e398612ba0274c14f2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 839df5100c814d6bb78dc67a8c834a49. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:50.491334 3676 raft_consensus.cc:2932] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } } }
I20240501 07:59:50.494661 3531 catalog_manager.cc:5484] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c reported cstate change: config changed from index -1 to 28, NON_VOTER 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:50.508829 3676 consensus_queue.cc:237] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } }
I20240501 07:59:50.510581 3939 raft_consensus.cc:1261] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20240501 07:59:50.511003 4668 consensus_queue.cc:1035] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240501 07:59:50.514511 4668 raft_consensus.cc:2932] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } } }
I20240501 07:59:50.514932 3939 raft_consensus.cc:2932] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } } }
W20240501 07:59:50.519141 3615 consensus_peers.cc:485] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad -> Peer 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134:36011): Couldn't send request to peer 6208df88dcc34b4daf3d27a574e53a95. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02f63ed2bd6345b686c74eb002bb306f. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:50.523550 3531 catalog_manager.cc:5484] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c reported cstate change: config changed from index -1 to 37, NON_VOTER 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } } }
I20240501 07:59:50.530220 3677 consensus_queue.cc:237] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.43, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } }
I20240501 07:59:50.532745 3806 raft_consensus.cc:1261] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 43. Preceding OpId from leader: term: 1 index: 44. (index mismatch)
I20240501 07:59:50.534098 5424 consensus_queue.cc:1035] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20240501 07:59:50.536500 5424 raft_consensus.cc:2932] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Committing config change with OpId 1.44: config changed from index -1 to 44, NON_VOTER da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } } }
I20240501 07:59:50.537331 3806 raft_consensus.cc:2932] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Committing config change with OpId 1.44: config changed from index -1 to 44, NON_VOTER da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } } }
W20240501 07:59:50.543996 3612 consensus_peers.cc:485] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad -> Peer da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133:40225): Couldn't send request to peer da103aa972ab49889fc0bce4abc56bf7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c0cda2d32534bb29a6a2538ac13368a. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:50.546308 3528 catalog_manager.cc:5484] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e reported cstate change: config changed from index -1 to 44, NON_VOTER da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } } }
I20240501 07:59:50.583330 3939 consensus_queue.cc:237] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } }
I20240501 07:59:50.585521 3677 raft_consensus.cc:1261] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 07:59:50.586100 5414 consensus_queue.cc:1035] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 07:59:50.589066 5414 raft_consensus.cc:2932] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } } }
W20240501 07:59:50.589587 3876 consensus_peers.cc:485] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c -> Peer d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132:37117): Couldn't send request to peer d8461c1ff1ed48b09047faeea87e9598. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31a0bc2519cd43eda2f2fc8b5a6e7d5b. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:50.592506 3528 catalog_manager.cc:5484] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c reported cstate change: config changed from index -1 to 28, NON_VOTER d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:50.597158 3677 raft_consensus.cc:2932] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } } }
I20240501 07:59:50.614406 3941 consensus_queue.cc:237] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 1.45, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } }
W20240501 07:59:50.615895 3875 consensus_peers.cc:485] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c -> Peer 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135:41559): Couldn't send request to peer 09d8fa5c011b4e398612ba0274c14f2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc31e13bb07f44ddb7b80282ee4c434b. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:50.617846 3677 raft_consensus.cc:1261] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 45. Preceding OpId from leader: term: 1 index: 46. (index mismatch)
I20240501 07:59:50.618534 5362 consensus_queue.cc:1035] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20240501 07:59:50.625059 5429 raft_consensus.cc:2932] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } } }
I20240501 07:59:50.629091 3528 catalog_manager.cc:5484] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c reported cstate change: config changed from index -1 to 46, NON_VOTER 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:50.640661 3677 raft_consensus.cc:2932] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } } }
I20240501 07:59:50.658946 3941 consensus_queue.cc:237] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } }
W20240501 07:59:50.665791 3877 consensus_peers.cc:485] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c -> Peer 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134:36011): Couldn't send request to peer 6208df88dcc34b4daf3d27a574e53a95. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c02a65bc31814809ae316ae5fdd07a6e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:50.666211 3808 raft_consensus.cc:1261] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240501 07:59:50.667379 5362 consensus_queue.cc:1035] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 37, Time since last communication: 0.000s
I20240501 07:59:50.671401 5362 raft_consensus.cc:2932] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index -1 to 39, NON_VOTER 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } } }
I20240501 07:59:50.679706 3528 catalog_manager.cc:5484] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c reported cstate change: config changed from index -1 to 39, NON_VOTER 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:50.695082 3808 raft_consensus.cc:2932] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index -1 to 39, NON_VOTER 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } } }
I20240501 07:59:50.758023 3806 consensus_queue.cc:237] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 1.49, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } }
W20240501 07:59:50.760047 3743 consensus_peers.cc:485] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e -> Peer da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133:40225): Couldn't send request to peer da103aa972ab49889fc0bce4abc56bf7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5432d22bf6dc4866bbc3d91f21a45c21. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:50.764540 3678 raft_consensus.cc:1261] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 50. (index mismatch)
I20240501 07:59:50.767832 5371 consensus_queue.cc:1035] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 48, Time since last communication: 0.000s
I20240501 07:59:50.772258 5371 raft_consensus.cc:2932] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Committing config change with OpId 1.50: config changed from index -1 to 50, NON_VOTER da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } } }
I20240501 07:59:50.777189 3678 raft_consensus.cc:2932] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Committing config change with OpId 1.50: config changed from index -1 to 50, NON_VOTER da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } } }
I20240501 07:59:50.780913 3531 catalog_manager.cc:5484] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: config changed from index -1 to 50, NON_VOTER da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } } }
I20240501 07:59:50.802217 3938 consensus_queue.cc:237] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } }
W20240501 07:59:50.804236 3875 consensus_peers.cc:485] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c -> Peer 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135:41559): Couldn't send request to peer 09d8fa5c011b4e398612ba0274c14f2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54a2b2f834494f29b306e80e48804a43. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:50.806699 3806 raft_consensus.cc:1261] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch)
I20240501 07:59:50.807948 5428 consensus_queue.cc:1035] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20240501 07:59:50.810731 5431 raft_consensus.cc:2932] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Committing config change with OpId 1.40: config changed from index -1 to 40, NON_VOTER 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } } }
I20240501 07:59:50.813844 3528 catalog_manager.cc:5484] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c reported cstate change: config changed from index -1 to 40, NON_VOTER 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:50.829777 3806 raft_consensus.cc:2932] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index -1 to 40, NON_VOTER 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } } }
I20240501 07:59:50.851675 5586 ts_tablet_manager.cc:902] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95: Initiating tablet copy from peer 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:50.853361 5586 tablet_copy_client.cc:308] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43191
I20240501 07:59:50.855158 3676 consensus_queue.cc:237] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 1.51, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } }
I20240501 07:59:50.858824 3806 raft_consensus.cc:1261] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 51. Preceding OpId from leader: term: 1 index: 52. (index mismatch)
I20240501 07:59:50.862313 5382 consensus_queue.cc:1035] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20240501 07:59:50.866160 3830 tablet_copy_service.cc:140] P 40c9ccd5f29c424eb845d670a6f7d21e: Received BeginTabletCopySession request for tablet ed30e32a4ba2483484b5bc5e3cfb44fd from peer 6208df88dcc34b4daf3d27a574e53a95 ({username='slave'} at 127.0.99.134:57811)
I20240501 07:59:50.866328 3830 tablet_copy_service.cc:161] P 40c9ccd5f29c424eb845d670a6f7d21e: Beginning new tablet copy session on tablet ed30e32a4ba2483484b5bc5e3cfb44fd from peer 6208df88dcc34b4daf3d27a574e53a95 at {username='slave'} at 127.0.99.134:57811: session id = 6208df88dcc34b4daf3d27a574e53a95-ed30e32a4ba2483484b5bc5e3cfb44fd
I20240501 07:59:50.867700 3830 tablet_copy_source_session.cc:205] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:50.869413 5593 ts_tablet_manager.cc:902] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e: Initiating tablet copy from peer 8635309522a045b1941fea81699d524c (127.0.99.131:39655)
I20240501 07:59:50.870230 5593 tablet_copy_client.cc:308] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39655
I20240501 07:59:50.870973 5382 raft_consensus.cc:2932] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Committing config change with OpId 1.52: config changed from index -1 to 52, NON_VOTER da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } } }
I20240501 07:59:50.872874 5586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed30e32a4ba2483484b5bc5e3cfb44fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.873461 3808 raft_consensus.cc:2932] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Committing config change with OpId 1.52: config changed from index -1 to 52, NON_VOTER da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } } }
I20240501 07:59:50.875525 3528 catalog_manager.cc:5484] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: config changed from index -1 to 52, NON_VOTER da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" committed_config { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:50.875945 3961 tablet_copy_service.cc:140] P 8635309522a045b1941fea81699d524c: Received BeginTabletCopySession request for tablet 839df5100c814d6bb78dc67a8c834a49 from peer 09d8fa5c011b4e398612ba0274c14f2e ({username='slave'} at 127.0.99.135:60945)
I20240501 07:59:50.876076 3961 tablet_copy_service.cc:161] P 8635309522a045b1941fea81699d524c: Beginning new tablet copy session on tablet 839df5100c814d6bb78dc67a8c834a49 from peer 09d8fa5c011b4e398612ba0274c14f2e at {username='slave'} at 127.0.99.135:60945: session id = 09d8fa5c011b4e398612ba0274c14f2e-839df5100c814d6bb78dc67a8c834a49
I20240501 07:59:50.877501 3961 tablet_copy_source_session.cc:205] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 07:59:50.878424 3612 consensus_peers.cc:485] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad -> Peer da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133:40225): Couldn't send request to peer da103aa972ab49889fc0bce4abc56bf7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1aa85993dc449b7b33ebccb4e907046. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:50.880033 5593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 839df5100c814d6bb78dc67a8c834a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.884716 5593 tablet_copy_client.cc:783] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:50.884960 5593 tablet_copy_client.cc:647] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:50.888522 5586 tablet_copy_client.cc:783] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:50.888732 5593 tablet_copy_client.cc:523] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:50.888926 5586 tablet_copy_client.cc:647] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:50.890933 5593 tablet_bootstrap.cc:492] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e: Bootstrap starting.
I20240501 07:59:50.895313 5604 ts_tablet_manager.cc:902] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95: Initiating tablet copy from peer 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:50.906688 5604 tablet_copy_client.cc:308] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42453
I20240501 07:59:50.911255 3699 tablet_copy_service.cc:140] P 014bc50d0c5a44bdbf49c0eb41fa93ad: Received BeginTabletCopySession request for tablet 02f63ed2bd6345b686c74eb002bb306f from peer 6208df88dcc34b4daf3d27a574e53a95 ({username='slave'} at 127.0.99.134:59099)
I20240501 07:59:50.911412 3699 tablet_copy_service.cc:161] P 014bc50d0c5a44bdbf49c0eb41fa93ad: Beginning new tablet copy session on tablet 02f63ed2bd6345b686c74eb002bb306f from peer 6208df88dcc34b4daf3d27a574e53a95 at {username='slave'} at 127.0.99.134:59099: session id = 6208df88dcc34b4daf3d27a574e53a95-02f63ed2bd6345b686c74eb002bb306f
I20240501 07:59:50.912762 3699 tablet_copy_source_session.cc:205] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:50.913967 5604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02f63ed2bd6345b686c74eb002bb306f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.918093 5604 tablet_copy_client.cc:783] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:50.921797 5586 tablet_copy_client.cc:523] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:50.921839 5604 tablet_copy_client.cc:647] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:50.924064 5586 tablet_bootstrap.cc:492] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95: Bootstrap starting.
I20240501 07:59:50.931232 5604 tablet_copy_client.cc:523] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:50.934065 5604 tablet_bootstrap.cc:492] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95: Bootstrap starting.
I20240501 07:59:50.946915 5586 log.cc:824] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:50.947715 5615 ts_tablet_manager.cc:902] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7: Initiating tablet copy from peer 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:50.952253 5615 tablet_copy_client.cc:308] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42453
I20240501 07:59:50.976224 5593 tablet_bootstrap.cc:492] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=34 ignored=0} inserts{seen=665 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:59:50.976658 5593 tablet_bootstrap.cc:492] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e: Bootstrap complete.
I20240501 07:59:50.976943 5593 ts_tablet_manager.cc:1367] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e: Time spent bootstrapping tablet: real 0.086s user 0.052s sys 0.001s
I20240501 07:59:50.977605 5593 raft_consensus.cc:348] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } }
I20240501 07:59:50.980262 5593 raft_consensus.cc:729] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09d8fa5c011b4e398612ba0274c14f2e, State: Initialized, Role: LEARNER
I20240501 07:59:50.980479 5593 consensus_queue.cc:260] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 1.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } }
I20240501 07:59:50.980979 5593 ts_tablet_manager.cc:1398] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:50.982192 3961 tablet_copy_service.cc:342] P 8635309522a045b1941fea81699d524c: Request end of tablet copy session 09d8fa5c011b4e398612ba0274c14f2e-839df5100c814d6bb78dc67a8c834a49 received from {username='slave'} at 127.0.99.135:60945
I20240501 07:59:50.982291 3961 tablet_copy_service.cc:433] P 8635309522a045b1941fea81699d524c: ending tablet copy session 09d8fa5c011b4e398612ba0274c14f2e-839df5100c814d6bb78dc67a8c834a49 on tablet 839df5100c814d6bb78dc67a8c834a49 with peer 09d8fa5c011b4e398612ba0274c14f2e
I20240501 07:59:50.999471 5630 ts_tablet_manager.cc:902] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598: Initiating tablet copy from peer 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:51.001189 5630 tablet_copy_client.cc:308] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43191
I20240501 07:59:51.006469 3699 tablet_copy_service.cc:140] P 014bc50d0c5a44bdbf49c0eb41fa93ad: Received BeginTabletCopySession request for tablet 7c0cda2d32534bb29a6a2538ac13368a from peer da103aa972ab49889fc0bce4abc56bf7 ({username='slave'} at 127.0.99.133:43635)
I20240501 07:59:51.006588 3699 tablet_copy_service.cc:161] P 014bc50d0c5a44bdbf49c0eb41fa93ad: Beginning new tablet copy session on tablet 7c0cda2d32534bb29a6a2538ac13368a from peer da103aa972ab49889fc0bce4abc56bf7 at {username='slave'} at 127.0.99.133:43635: session id = da103aa972ab49889fc0bce4abc56bf7-7c0cda2d32534bb29a6a2538ac13368a
I20240501 07:59:51.007615 3699 tablet_copy_source_session.cc:205] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:51.012754 5615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0cda2d32534bb29a6a2538ac13368a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.021411 5629 raft_consensus.cc:482] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:51.021704 5629 raft_consensus.cc:504] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } }
I20240501 07:59:51.022135 5615 tablet_copy_client.cc:783] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:51.024669 5629 leader_election.cc:288] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132:37117)
I20240501 07:59:51.025405 5615 tablet_copy_client.cc:647] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:51.028000 4893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a2915d4efd4d48a1d41df5ecf2f4d7" candidate_uuid: "09d8fa5c011b4e398612ba0274c14f2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8461c1ff1ed48b09047faeea87e9598" is_pre_election: true
I20240501 07:59:51.028263 4893 raft_consensus.cc:2454] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09d8fa5c011b4e398612ba0274c14f2e in term 1.
I20240501 07:59:51.028784 5223 leader_election.cc:302] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 09d8fa5c011b4e398612ba0274c14f2e, d8461c1ff1ed48b09047faeea87e9598; no voters:
I20240501 07:59:51.030042 5615 tablet_copy_client.cc:523] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:51.030090 5629 raft_consensus.cc:2781] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:51.030221 5629 raft_consensus.cc:482] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:51.030323 5629 raft_consensus.cc:3037] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:51.031898 5629 raft_consensus.cc:504] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } }
I20240501 07:59:51.032284 5629 leader_election.cc:288] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [CANDIDATE]: Term 2 election: Requested vote from peers d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132:37117)
I20240501 07:59:51.032651 4893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a2915d4efd4d48a1d41df5ecf2f4d7" candidate_uuid: "09d8fa5c011b4e398612ba0274c14f2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8461c1ff1ed48b09047faeea87e9598"
I20240501 07:59:51.032783 4893 raft_consensus.cc:3037] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:51.032837 5626 ts_tablet_manager.cc:902] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7: Initiating tablet copy from peer 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:51.033421 5626 tablet_copy_client.cc:308] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43191
I20240501 07:59:51.034991 4893 raft_consensus.cc:2454] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09d8fa5c011b4e398612ba0274c14f2e in term 2.
I20240501 07:59:51.036118 5223 leader_election.cc:302] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 09d8fa5c011b4e398612ba0274c14f2e, d8461c1ff1ed48b09047faeea87e9598; no voters:
I20240501 07:59:51.036399 5629 raft_consensus.cc:2781] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:51.036621 5629 raft_consensus.cc:686] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 2 LEADER]: Becoming Leader. State: Replica: 09d8fa5c011b4e398612ba0274c14f2e, State: Running, Role: LEADER
I20240501 07:59:51.036993 5629 consensus_queue.cc:237] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } }
I20240501 07:59:51.039736 3528 catalog_manager.cc:5484] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "09d8fa5c011b4e398612ba0274c14f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:51.042999 5615 tablet_bootstrap.cc:492] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7: Bootstrap starting.
I20240501 07:59:51.063249 3830 tablet_copy_service.cc:140] P 40c9ccd5f29c424eb845d670a6f7d21e: Received BeginTabletCopySession request for tablet f75acaf04c12462aa4b3fa7e019776d4 from peer d8461c1ff1ed48b09047faeea87e9598 ({username='slave'} at 127.0.99.132:56705)
I20240501 07:59:51.063375 3830 tablet_copy_service.cc:161] P 40c9ccd5f29c424eb845d670a6f7d21e: Beginning new tablet copy session on tablet f75acaf04c12462aa4b3fa7e019776d4 from peer d8461c1ff1ed48b09047faeea87e9598 at {username='slave'} at 127.0.99.132:56705: session id = d8461c1ff1ed48b09047faeea87e9598-f75acaf04c12462aa4b3fa7e019776d4
I20240501 07:59:51.064324 3830 tablet_copy_source_session.cc:205] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:51.067405 5630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75acaf04c12462aa4b3fa7e019776d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.072855 5630 tablet_copy_client.cc:783] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:51.073154 5630 tablet_copy_client.cc:647] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:51.074440 3829 tablet_copy_service.cc:140] P 40c9ccd5f29c424eb845d670a6f7d21e: Received BeginTabletCopySession request for tablet 393b4eef45ff435abd0fde4aec22da88 from peer da103aa972ab49889fc0bce4abc56bf7 ({username='slave'} at 127.0.99.133:35275)
I20240501 07:59:51.074544 3829 tablet_copy_service.cc:161] P 40c9ccd5f29c424eb845d670a6f7d21e: Beginning new tablet copy session on tablet 393b4eef45ff435abd0fde4aec22da88 from peer da103aa972ab49889fc0bce4abc56bf7 at {username='slave'} at 127.0.99.133:35275: session id = da103aa972ab49889fc0bce4abc56bf7-393b4eef45ff435abd0fde4aec22da88
I20240501 07:59:51.074996 5630 tablet_copy_client.cc:523] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:51.075539 3829 tablet_copy_source_session.cc:205] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:51.076517 5630 tablet_bootstrap.cc:492] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598: Bootstrap starting.
I20240501 07:59:51.077567 5626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393b4eef45ff435abd0fde4aec22da88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.078184 5615 log.cc:824] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:51.081804 5626 tablet_copy_client.cc:783] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:51.082116 5626 tablet_copy_client.cc:647] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:51.083462 5626 tablet_copy_client.cc:523] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:51.085283 5626 tablet_bootstrap.cc:492] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7: Bootstrap starting.
I20240501 07:59:51.093014 5593 ts_tablet_manager.cc:902] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e: Initiating tablet copy from peer 8635309522a045b1941fea81699d524c (127.0.99.131:39655)
I20240501 07:59:51.093590 5593 tablet_copy_client.cc:308] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39655
I20240501 07:59:51.094769 3961 tablet_copy_service.cc:140] P 8635309522a045b1941fea81699d524c: Received BeginTabletCopySession request for tablet cc31e13bb07f44ddb7b80282ee4c434b from peer 09d8fa5c011b4e398612ba0274c14f2e ({username='slave'} at 127.0.99.135:60945)
I20240501 07:59:51.094883 3961 tablet_copy_service.cc:161] P 8635309522a045b1941fea81699d524c: Beginning new tablet copy session on tablet cc31e13bb07f44ddb7b80282ee4c434b from peer 09d8fa5c011b4e398612ba0274c14f2e at {username='slave'} at 127.0.99.135:60945: session id = 09d8fa5c011b4e398612ba0274c14f2e-cc31e13bb07f44ddb7b80282ee4c434b
I20240501 07:59:51.095899 3961 tablet_copy_source_session.cc:205] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:51.097348 5593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc31e13bb07f44ddb7b80282ee4c434b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.100201 5593 tablet_copy_client.cc:783] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:51.100677 5593 tablet_copy_client.cc:647] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:51.112442 5593 tablet_copy_client.cc:523] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:51.113294 5586 tablet_bootstrap.cc:492] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95: Bootstrap replayed 1/1 log segments. Stats: ops{read=64 overwritten=0 applied=64 ignored=0} inserts{seen=736 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:51.113633 5586 tablet_bootstrap.cc:492] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95: Bootstrap complete.
I20240501 07:59:51.114099 5586 ts_tablet_manager.cc:1367] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95: Time spent bootstrapping tablet: real 0.190s user 0.084s sys 0.002s
I20240501 07:59:51.114740 5593 tablet_bootstrap.cc:492] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e: Bootstrap starting.
I20240501 07:59:51.123842 5586 raft_consensus.cc:348] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } }
I20240501 07:59:51.124305 5586 raft_consensus.cc:729] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6208df88dcc34b4daf3d27a574e53a95, State: Initialized, Role: LEARNER
I20240501 07:59:51.124794 5586 consensus_queue.cc:260] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 64, Last appended: 1.64, Last appended by leader: 64, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } }
I20240501 07:59:51.135183 5586 ts_tablet_manager.cc:1398] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20240501 07:59:51.138478 3829 tablet_copy_service.cc:342] P 40c9ccd5f29c424eb845d670a6f7d21e: Request end of tablet copy session 6208df88dcc34b4daf3d27a574e53a95-ed30e32a4ba2483484b5bc5e3cfb44fd received from {username='slave'} at 127.0.99.134:57811
I20240501 07:59:51.138579 3829 tablet_copy_service.cc:433] P 40c9ccd5f29c424eb845d670a6f7d21e: ending tablet copy session 6208df88dcc34b4daf3d27a574e53a95-ed30e32a4ba2483484b5bc5e3cfb44fd on tablet ed30e32a4ba2483484b5bc5e3cfb44fd with peer 6208df88dcc34b4daf3d27a574e53a95
I20240501 07:59:51.196417 5666 ts_tablet_manager.cc:902] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598: Initiating tablet copy from peer 8635309522a045b1941fea81699d524c (127.0.99.131:39655)
I20240501 07:59:51.197337 5666 tablet_copy_client.cc:308] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39655
I20240501 07:59:51.213121 5672 ts_tablet_manager.cc:902] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7: Initiating tablet copy from peer 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:51.213809 5672 tablet_copy_client.cc:308] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43191
I20240501 07:59:51.214849 3829 tablet_copy_service.cc:140] P 40c9ccd5f29c424eb845d670a6f7d21e: Received BeginTabletCopySession request for tablet 5432d22bf6dc4866bbc3d91f21a45c21 from peer da103aa972ab49889fc0bce4abc56bf7 ({username='slave'} at 127.0.99.133:35275)
I20240501 07:59:51.214973 3829 tablet_copy_service.cc:161] P 40c9ccd5f29c424eb845d670a6f7d21e: Beginning new tablet copy session on tablet 5432d22bf6dc4866bbc3d91f21a45c21 from peer da103aa972ab49889fc0bce4abc56bf7 at {username='slave'} at 127.0.99.133:35275: session id = da103aa972ab49889fc0bce4abc56bf7-5432d22bf6dc4866bbc3d91f21a45c21
I20240501 07:59:51.215958 3829 tablet_copy_source_session.cc:205] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:51.218341 3961 tablet_copy_service.cc:140] P 8635309522a045b1941fea81699d524c: Received BeginTabletCopySession request for tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b from peer d8461c1ff1ed48b09047faeea87e9598 ({username='slave'} at 127.0.99.132:50465)
I20240501 07:59:51.218444 3961 tablet_copy_service.cc:161] P 8635309522a045b1941fea81699d524c: Beginning new tablet copy session on tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b from peer d8461c1ff1ed48b09047faeea87e9598 at {username='slave'} at 127.0.99.132:50465: session id = d8461c1ff1ed48b09047faeea87e9598-31a0bc2519cd43eda2f2fc8b5a6e7d5b
I20240501 07:59:51.219414 3961 tablet_copy_source_session.cc:205] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:51.222388 5672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5432d22bf6dc4866bbc3d91f21a45c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.224455 5672 tablet_copy_client.cc:783] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:51.224596 5672 tablet_copy_client.cc:647] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:51.232986 5666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.240708 5666 tablet_copy_client.cc:783] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:51.240983 5666 tablet_copy_client.cc:647] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:51.242295 5672 tablet_copy_client.cc:523] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:51.242502 5666 tablet_copy_client.cc:523] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:51.243674 5672 tablet_bootstrap.cc:492] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7: Bootstrap starting.
I20240501 07:59:51.243794 5666 tablet_bootstrap.cc:492] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598: Bootstrap starting.
I20240501 07:59:51.254056 5681 ts_tablet_manager.cc:902] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e: Initiating tablet copy from peer 8635309522a045b1941fea81699d524c (127.0.99.131:39655)
I20240501 07:59:51.254774 5681 tablet_copy_client.cc:308] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39655
I20240501 07:59:51.257386 3961 tablet_copy_service.cc:140] P 8635309522a045b1941fea81699d524c: Received BeginTabletCopySession request for tablet 54a2b2f834494f29b306e80e48804a43 from peer 09d8fa5c011b4e398612ba0274c14f2e ({username='slave'} at 127.0.99.135:60945)
I20240501 07:59:51.257500 3961 tablet_copy_service.cc:161] P 8635309522a045b1941fea81699d524c: Beginning new tablet copy session on tablet 54a2b2f834494f29b306e80e48804a43 from peer 09d8fa5c011b4e398612ba0274c14f2e at {username='slave'} at 127.0.99.135:60945: session id = 09d8fa5c011b4e398612ba0274c14f2e-54a2b2f834494f29b306e80e48804a43
I20240501 07:59:51.258482 3961 tablet_copy_source_session.cc:205] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:51.275600 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a2b2f834494f29b306e80e48804a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.278580 5279 raft_consensus.cc:1203] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.46] Dedup: 1.35->[1.36-1.46]
I20240501 07:59:51.287278 5615 tablet_bootstrap.cc:492] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7: Bootstrap replayed 1/1 log segments. Stats: ops{read=55 overwritten=0 applied=55 ignored=0} inserts{seen=740 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:51.291632 5681 tablet_copy_client.cc:783] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:51.291901 5681 tablet_copy_client.cc:647] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:51.294000 5681 tablet_copy_client.cc:523] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:51.294356 5615 tablet_bootstrap.cc:492] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7: Bootstrap complete.
I20240501 07:59:51.296731 5615 ts_tablet_manager.cc:1367] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7: Time spent bootstrapping tablet: real 0.254s user 0.071s sys 0.007s
I20240501 07:59:51.309072 5615 raft_consensus.cc:348] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } }
I20240501 07:59:51.309541 5615 raft_consensus.cc:729] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da103aa972ab49889fc0bce4abc56bf7, State: Initialized, Role: LEARNER
I20240501 07:59:51.310070 5615 consensus_queue.cc:260] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 55, Last appended: 1.55, Last appended by leader: 55, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } }
I20240501 07:59:51.311240 5615 ts_tablet_manager.cc:1398] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240501 07:59:51.318305 5700 ts_tablet_manager.cc:902] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7: Initiating tablet copy from peer 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:51.319000 5700 tablet_copy_client.cc:308] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42453
I20240501 07:59:51.320582 3699 tablet_copy_service.cc:140] P 014bc50d0c5a44bdbf49c0eb41fa93ad: Received BeginTabletCopySession request for tablet f1aa85993dc449b7b33ebccb4e907046 from peer da103aa972ab49889fc0bce4abc56bf7 ({username='slave'} at 127.0.99.133:43635)
I20240501 07:59:51.320703 3699 tablet_copy_service.cc:161] P 014bc50d0c5a44bdbf49c0eb41fa93ad: Beginning new tablet copy session on tablet f1aa85993dc449b7b33ebccb4e907046 from peer da103aa972ab49889fc0bce4abc56bf7 at {username='slave'} at 127.0.99.133:43635: session id = da103aa972ab49889fc0bce4abc56bf7-f1aa85993dc449b7b33ebccb4e907046
I20240501 07:59:51.321633 3699 tablet_copy_source_session.cc:205] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:51.322343 3699 tablet_copy_service.cc:342] P 014bc50d0c5a44bdbf49c0eb41fa93ad: Request end of tablet copy session da103aa972ab49889fc0bce4abc56bf7-7c0cda2d32534bb29a6a2538ac13368a received from {username='slave'} at 127.0.99.133:43635
I20240501 07:59:51.322427 3699 tablet_copy_service.cc:433] P 014bc50d0c5a44bdbf49c0eb41fa93ad: ending tablet copy session da103aa972ab49889fc0bce4abc56bf7-7c0cda2d32534bb29a6a2538ac13368a on tablet 7c0cda2d32534bb29a6a2538ac13368a with peer da103aa972ab49889fc0bce4abc56bf7
I20240501 07:59:51.323776 5586 ts_tablet_manager.cc:902] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95: Initiating tablet copy from peer 8635309522a045b1941fea81699d524c (127.0.99.131:39655)
I20240501 07:59:51.324205 5586 tablet_copy_client.cc:308] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39655
I20240501 07:59:51.324595 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1aa85993dc449b7b33ebccb4e907046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.328903 3961 tablet_copy_service.cc:140] P 8635309522a045b1941fea81699d524c: Received BeginTabletCopySession request for tablet c02a65bc31814809ae316ae5fdd07a6e from peer 6208df88dcc34b4daf3d27a574e53a95 ({username='slave'} at 127.0.99.134:53833)
I20240501 07:59:51.329023 3961 tablet_copy_service.cc:161] P 8635309522a045b1941fea81699d524c: Beginning new tablet copy session on tablet c02a65bc31814809ae316ae5fdd07a6e from peer 6208df88dcc34b4daf3d27a574e53a95 at {username='slave'} at 127.0.99.134:53833: session id = 6208df88dcc34b4daf3d27a574e53a95-c02a65bc31814809ae316ae5fdd07a6e
I20240501 07:59:51.329111 5700 tablet_copy_client.cc:783] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:51.329273 5700 tablet_copy_client.cc:647] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:51.329885 3961 tablet_copy_source_session.cc:205] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:51.331030 5586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02a65bc31814809ae316ae5fdd07a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.333232 5586 tablet_copy_client.cc:783] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:51.333375 5586 tablet_copy_client.cc:647] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:51.337272 5700 tablet_copy_client.cc:523] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:51.341956 5604 tablet_bootstrap.cc:492] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95: Bootstrap replayed 1/1 log segments. Stats: ops{read=54 overwritten=0 applied=54 ignored=0} inserts{seen=685 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:51.342492 5604 tablet_bootstrap.cc:492] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95: Bootstrap complete.
I20240501 07:59:51.342839 5604 ts_tablet_manager.cc:1367] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95: Time spent bootstrapping tablet: real 0.409s user 0.062s sys 0.008s
I20240501 07:59:51.342880 5700 tablet_bootstrap.cc:492] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7: Bootstrap starting.
I20240501 07:59:51.343806 5604 raft_consensus.cc:348] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } }
I20240501 07:59:51.345927 5604 raft_consensus.cc:729] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6208df88dcc34b4daf3d27a574e53a95, State: Initialized, Role: LEARNER
I20240501 07:59:51.346328 5604 consensus_queue.cc:260] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 54, Last appended: 1.54, Last appended by leader: 54, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } }
I20240501 07:59:51.348481 5604 ts_tablet_manager.cc:1398] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:59:51.350337 3699 tablet_copy_service.cc:342] P 014bc50d0c5a44bdbf49c0eb41fa93ad: Request end of tablet copy session 6208df88dcc34b4daf3d27a574e53a95-02f63ed2bd6345b686c74eb002bb306f received from {username='slave'} at 127.0.99.134:59099
I20240501 07:59:51.350430 3699 tablet_copy_service.cc:433] P 014bc50d0c5a44bdbf49c0eb41fa93ad: ending tablet copy session 6208df88dcc34b4daf3d27a574e53a95-02f63ed2bd6345b686c74eb002bb306f on tablet 02f63ed2bd6345b686c74eb002bb306f with peer 6208df88dcc34b4daf3d27a574e53a95
I20240501 07:59:51.351814 5626 tablet_bootstrap.cc:492] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7: Bootstrap replayed 1/1 log segments. Stats: ops{read=68 overwritten=0 applied=68 ignored=0} inserts{seen=753 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:51.358336 5626 tablet_bootstrap.cc:492] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7: Bootstrap complete.
I20240501 07:59:51.361346 5626 ts_tablet_manager.cc:1367] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7: Time spent bootstrapping tablet: real 0.276s user 0.073s sys 0.005s
I20240501 07:59:51.362182 5626 raft_consensus.cc:348] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } }
I20240501 07:59:51.362453 5626 raft_consensus.cc:729] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da103aa972ab49889fc0bce4abc56bf7, State: Initialized, Role: LEARNER
I20240501 07:59:51.362602 5626 consensus_queue.cc:260] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 68, Last appended: 1.68, Last appended by leader: 68, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } }
I20240501 07:59:51.362973 5626 ts_tablet_manager.cc:1398] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:51.363481 3829 tablet_copy_service.cc:342] P 40c9ccd5f29c424eb845d670a6f7d21e: Request end of tablet copy session da103aa972ab49889fc0bce4abc56bf7-393b4eef45ff435abd0fde4aec22da88 received from {username='slave'} at 127.0.99.133:35275
I20240501 07:59:51.363564 3829 tablet_copy_service.cc:433] P 40c9ccd5f29c424eb845d670a6f7d21e: ending tablet copy session da103aa972ab49889fc0bce4abc56bf7-393b4eef45ff435abd0fde4aec22da88 on tablet 393b4eef45ff435abd0fde4aec22da88 with peer da103aa972ab49889fc0bce4abc56bf7
I20240501 07:59:51.364152 5157 raft_consensus.cc:1203] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.36->[1.37-1.67] Dedup: 1.54->[1.55-1.67]
I20240501 07:59:51.368448 5586 tablet_copy_client.cc:523] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:51.369901 5586 tablet_bootstrap.cc:492] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95: Bootstrap starting.
I20240501 07:59:51.375288 5593 tablet_bootstrap.cc:492] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=58 overwritten=0 applied=57 ignored=0} inserts{seen=688 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:59:51.375756 5593 tablet_bootstrap.cc:492] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e: Bootstrap complete.
I20240501 07:59:51.376019 5593 ts_tablet_manager.cc:1367] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e: Time spent bootstrapping tablet: real 0.261s user 0.069s sys 0.000s
I20240501 07:59:51.376580 5593 raft_consensus.cc:348] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } }
I20240501 07:59:51.377275 5593 raft_consensus.cc:729] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09d8fa5c011b4e398612ba0274c14f2e, State: Initialized, Role: LEARNER
I20240501 07:59:51.377460 5593 consensus_queue.cc:260] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 57, Last appended: 1.58, Last appended by leader: 58, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } }
I20240501 07:59:51.379527 5593 ts_tablet_manager.cc:1398] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:51.381578 3961 tablet_copy_service.cc:342] P 8635309522a045b1941fea81699d524c: Request end of tablet copy session 09d8fa5c011b4e398612ba0274c14f2e-cc31e13bb07f44ddb7b80282ee4c434b received from {username='slave'} at 127.0.99.135:60945
I20240501 07:59:51.381667 3961 tablet_copy_service.cc:433] P 8635309522a045b1941fea81699d524c: ending tablet copy session 09d8fa5c011b4e398612ba0274c14f2e-cc31e13bb07f44ddb7b80282ee4c434b on tablet cc31e13bb07f44ddb7b80282ee4c434b with peer 09d8fa5c011b4e398612ba0274c14f2e
I20240501 07:59:51.383337 5026 raft_consensus.cc:1203] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.43->[1.44-1.68] Dedup: 1.55->[1.56-1.68]
I20240501 07:59:51.385499 5681 tablet_bootstrap.cc:492] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e: Bootstrap starting.
I20240501 07:59:51.388294 5431 raft_consensus.cc:1050] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c: attempting to promote NON_VOTER 09d8fa5c011b4e398612ba0274c14f2e to VOTER
I20240501 07:59:51.388782 5431 consensus_queue.cc:237] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 1.47, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } }
I20240501 07:59:51.390535 5279 raft_consensus.cc:1261] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 LEARNER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 1 index: 48. (index mismatch)
I20240501 07:59:51.391148 5429 consensus_queue.cc:1035] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 46, Time since last communication: 0.000s
I20240501 07:59:51.395509 5429 raft_consensus.cc:2932] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Committing config change with OpId 1.48: config changed from index 28 to 48, 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:51.398115 5279 raft_consensus.cc:2932] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Committing config change with OpId 1.48: config changed from index 28 to 48, 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:51.401309 3528 catalog_manager.cc:5484] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e reported cstate change: config changed from index 28 to 48, 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:51.411734 5672 tablet_bootstrap.cc:492] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7: Bootstrap replayed 1/1 log segments. Stats: ops{read=60 overwritten=0 applied=60 ignored=0} inserts{seen=745 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:51.412050 5672 tablet_bootstrap.cc:492] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7: Bootstrap complete.
I20240501 07:59:51.412240 5672 ts_tablet_manager.cc:1367] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7: Time spent bootstrapping tablet: real 0.169s user 0.063s sys 0.004s
I20240501 07:59:51.412762 5672 raft_consensus.cc:348] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } }
I20240501 07:59:51.412894 5672 raft_consensus.cc:729] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da103aa972ab49889fc0bce4abc56bf7, State: Initialized, Role: LEARNER
I20240501 07:59:51.413012 5672 consensus_queue.cc:260] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 60, Last appended: 1.60, Last appended by leader: 60, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } }
I20240501 07:59:51.416224 3678 raft_consensus.cc:1261] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 1 index: 48. (index mismatch)
I20240501 07:59:51.419593 5672 ts_tablet_manager.cc:1398] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 07:59:51.420296 3829 tablet_copy_service.cc:342] P 40c9ccd5f29c424eb845d670a6f7d21e: Request end of tablet copy session da103aa972ab49889fc0bce4abc56bf7-5432d22bf6dc4866bbc3d91f21a45c21 received from {username='slave'} at 127.0.99.133:35275
I20240501 07:59:51.420393 3829 tablet_copy_service.cc:433] P 40c9ccd5f29c424eb845d670a6f7d21e: ending tablet copy session da103aa972ab49889fc0bce4abc56bf7-5432d22bf6dc4866bbc3d91f21a45c21 on tablet 5432d22bf6dc4866bbc3d91f21a45c21 with peer da103aa972ab49889fc0bce4abc56bf7
I20240501 07:59:51.425045 5157 raft_consensus.cc:1203] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.35->[1.36-1.77] Dedup: 1.64->[1.65-1.77]
I20240501 07:59:51.434408 5666 tablet_bootstrap.cc:492] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=671 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:51.437039 5431 consensus_queue.cc:1035] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 46, Time since last communication: 0.000s
I20240501 07:59:51.437666 5409 raft_consensus.cc:1050] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e: attempting to promote NON_VOTER 6208df88dcc34b4daf3d27a574e53a95 to VOTER
I20240501 07:59:51.438104 5409 consensus_queue.cc:237] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } }
I20240501 07:59:51.439534 3940 raft_consensus.cc:1261] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240501 07:59:51.439944 4623 consensus_queue.cc:1035] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 77, Time since last communication: 0.000s
I20240501 07:59:51.440264 5157 raft_consensus.cc:1261] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95 [term 1 LEARNER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240501 07:59:51.440650 4623 consensus_queue.cc:1035] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 77, Time since last communication: 0.000s
I20240501 07:59:51.442145 4623 raft_consensus.cc:2932] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index 36 to 79, 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:51.442359 5666 tablet_bootstrap.cc:492] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598: Bootstrap complete.
I20240501 07:59:51.442577 5666 ts_tablet_manager.cc:1367] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598: Time spent bootstrapping tablet: real 0.199s user 0.054s sys 0.003s
I20240501 07:59:51.442819 5157 raft_consensus.cc:2932] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index 36 to 79, 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:51.443218 5666 raft_consensus.cc:348] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } }
I20240501 07:59:51.443337 5666 raft_consensus.cc:729] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8461c1ff1ed48b09047faeea87e9598, State: Initialized, Role: LEARNER
I20240501 07:59:51.443450 5666 consensus_queue.cc:260] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } }
I20240501 07:59:51.443826 5666 ts_tablet_manager.cc:1398] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:51.445576 3528 catalog_manager.cc:5484] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95 reported cstate change: config changed from index 36 to 79, 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:51.447726 3961 tablet_copy_service.cc:342] P 8635309522a045b1941fea81699d524c: Request end of tablet copy session d8461c1ff1ed48b09047faeea87e9598-31a0bc2519cd43eda2f2fc8b5a6e7d5b received from {username='slave'} at 127.0.99.132:50465
I20240501 07:59:51.447813 3961 tablet_copy_service.cc:433] P 8635309522a045b1941fea81699d524c: ending tablet copy session d8461c1ff1ed48b09047faeea87e9598-31a0bc2519cd43eda2f2fc8b5a6e7d5b on tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b with peer d8461c1ff1ed48b09047faeea87e9598
I20240501 07:59:51.451682 5430 raft_consensus.cc:1050] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad: attempting to promote NON_VOTER da103aa972ab49889fc0bce4abc56bf7 to VOTER
I20240501 07:59:51.451879 3808 consensus_queue.cc:237] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } }
I20240501 07:59:51.452086 5430 consensus_queue.cc:237] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } }
I20240501 07:59:51.452894 5157 raft_consensus.cc:1261] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95 [term 1 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240501 07:59:51.453285 4623 consensus_queue.cc:1035] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240501 07:59:51.454566 5026 raft_consensus.cc:1261] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7 [term 1 LEARNER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240501 07:59:51.454715 3809 raft_consensus.cc:1261] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240501 07:59:51.455072 5424 consensus_queue.cc:1035] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 68, Time since last communication: 0.000s
I20240501 07:59:51.455833 5424 consensus_queue.cc:1035] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 68, Time since last communication: 0.000s
I20240501 07:59:51.456483 5410 raft_consensus.cc:2932] T ed30e32a4ba2483484b5bc5e3cfb44fd P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index 79 to 80, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:51.457531 5424 raft_consensus.cc:2932] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index 44 to 70, da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.457513 5157 raft_consensus.cc:2932] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 79 to 80, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:51.458354 3523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ed30e32a4ba2483484b5bc5e3cfb44fd with cas_config_opid_index 79: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:51.460052 3528 catalog_manager.cc:5484] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: config changed from index 44 to 70, da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:51.460923 3809 raft_consensus.cc:2932] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 44 to 70, da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.461517 5026 raft_consensus.cc:2932] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 44 to 70, da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.462224 3940 raft_consensus.cc:2932] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index 36 to 79, 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:51.462745 3534 catalog_manager.cc:5484] T ed30e32a4ba2483484b5bc5e3cfb44fd P 6208df88dcc34b4daf3d27a574e53a95 reported cstate change: config changed from index 79 to 80, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:51.471429 3678 consensus_queue.cc:237] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } }
I20240501 07:59:51.473073 5026 raft_consensus.cc:1261] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240501 07:59:51.473484 5430 consensus_queue.cc:1035] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 70, Time since last communication: 0.000s
I20240501 07:59:51.474133 3678 raft_consensus.cc:2932] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Committing config change with OpId 1.48: config changed from index 28 to 48, 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:51.475983 3920 tablet_service.cc:1514] Processing DeleteTablet for tablet ed30e32a4ba2483484b5bc5e3cfb44fd with delete_type TABLET_DATA_TOMBSTONED (TS 8635309522a045b1941fea81699d524c not found in new config with opid_index 80) from {username='slave'} at 127.0.0.1:53920
I20240501 07:59:51.476634 5730 tablet_replica.cc:330] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c: stopping tablet replica
I20240501 07:59:51.476644 4668 raft_consensus.cc:2932] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index 70 to 72, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.477032 5730 raft_consensus.cc:2229] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:51.478186 3523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7c0cda2d32534bb29a6a2538ac13368a with cas_config_opid_index 70: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:51.479611 3528 catalog_manager.cc:5484] T 7c0cda2d32534bb29a6a2538ac13368a P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: config changed from index 70 to 72, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:51.480561 5026 raft_consensus.cc:2932] T 7c0cda2d32534bb29a6a2538ac13368a P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 70 to 72, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.482447 3790 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c0cda2d32534bb29a6a2538ac13368a with delete_type TABLET_DATA_TOMBSTONED (TS 40c9ccd5f29c424eb845d670a6f7d21e not found in new config with opid_index 72) from {username='slave'} at 127.0.0.1:43228
I20240501 07:59:51.493139 5731 tablet_replica.cc:330] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e: stopping tablet replica
I20240501 07:59:51.493582 5731 raft_consensus.cc:2229] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:51.493870 5731 raft_consensus.cc:2258] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:51.495698 5731 ts_tablet_manager.cc:1875] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:51.496949 5730 raft_consensus.cc:2258] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:51.505344 5731 ts_tablet_manager.cc:1888] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.71
I20240501 07:59:51.505635 5731 log.cc:1196] T 7c0cda2d32534bb29a6a2538ac13368a P 40c9ccd5f29c424eb845d670a6f7d21e: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7c0cda2d32534bb29a6a2538ac13368a
I20240501 07:59:51.509959 3523 catalog_manager.cc:4830] TS 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191): tablet 7c0cda2d32534bb29a6a2538ac13368a (table rebalance_test_table_2 [id=712e7be85ad54a6ebf02a160dca63546]) successfully deleted
I20240501 07:59:51.515265 5586 tablet_bootstrap.cc:492] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95: Bootstrap replayed 1/1 log segments. Stats: ops{read=50 overwritten=0 applied=50 ignored=0} inserts{seen=722 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:51.515841 5586 tablet_bootstrap.cc:492] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95: Bootstrap complete.
I20240501 07:59:51.516186 5586 ts_tablet_manager.cc:1367] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95: Time spent bootstrapping tablet: real 0.146s user 0.060s sys 0.003s
I20240501 07:59:51.516817 5586 raft_consensus.cc:348] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } }
I20240501 07:59:51.517030 5586 raft_consensus.cc:729] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6208df88dcc34b4daf3d27a574e53a95, State: Initialized, Role: LEARNER
I20240501 07:59:51.517189 5586 consensus_queue.cc:260] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 50, Last appended: 1.50, Last appended by leader: 50, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } }
I20240501 07:59:51.519652 5586 ts_tablet_manager.cc:1398] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:51.522732 3961 tablet_copy_service.cc:342] P 8635309522a045b1941fea81699d524c: Request end of tablet copy session 6208df88dcc34b4daf3d27a574e53a95-c02a65bc31814809ae316ae5fdd07a6e received from {username='slave'} at 127.0.99.134:53833
I20240501 07:59:51.522825 3961 tablet_copy_service.cc:433] P 8635309522a045b1941fea81699d524c: ending tablet copy session 6208df88dcc34b4daf3d27a574e53a95-c02a65bc31814809ae316ae5fdd07a6e on tablet c02a65bc31814809ae316ae5fdd07a6e with peer 6208df88dcc34b4daf3d27a574e53a95
I20240501 07:59:51.530620 5730 ts_tablet_manager.cc:1875] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:51.545221 5026 raft_consensus.cc:1203] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.41->[1.42-1.73] Dedup: 1.68->[1.69-1.73]
I20240501 07:59:51.548432 5730 ts_tablet_manager.cc:1888] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.79
I20240501 07:59:51.548650 5730 log.cc:1196] T ed30e32a4ba2483484b5bc5e3cfb44fd P 8635309522a045b1941fea81699d524c: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ed30e32a4ba2483484b5bc5e3cfb44fd
I20240501 07:59:51.550148 3523 catalog_manager.cc:4830] TS 8635309522a045b1941fea81699d524c (127.0.99.131:39655): tablet ed30e32a4ba2483484b5bc5e3cfb44fd (table rebalance_test_table_2 [id=712e7be85ad54a6ebf02a160dca63546]) successfully deleted
I20240501 07:59:51.569254 4623 raft_consensus.cc:1050] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e: attempting to promote NON_VOTER da103aa972ab49889fc0bce4abc56bf7 to VOTER
I20240501 07:59:51.574389 5700 tablet_bootstrap.cc:492] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7: Bootstrap replayed 1/1 log segments. Stats: ops{read=65 overwritten=0 applied=65 ignored=0} inserts{seen=750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:51.575066 5371 raft_consensus.cc:1050] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e: attempting to promote NON_VOTER da103aa972ab49889fc0bce4abc56bf7 to VOTER
I20240501 07:59:51.575440 5371 consensus_queue.cc:237] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.75, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } }
I20240501 07:59:51.575896 5700 tablet_bootstrap.cc:492] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7: Bootstrap complete.
I20240501 07:59:51.576161 5700 ts_tablet_manager.cc:1367] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7: Time spent bootstrapping tablet: real 0.233s user 0.077s sys 0.000s
I20240501 07:59:51.576645 5025 raft_consensus.cc:1261] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7 [term 1 LEARNER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240501 07:59:51.576712 5700 raft_consensus.cc:348] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } }
I20240501 07:59:51.576869 5700 raft_consensus.cc:729] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da103aa972ab49889fc0bce4abc56bf7, State: Initialized, Role: LEARNER
I20240501 07:59:51.576974 5700 consensus_queue.cc:260] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 65, Last appended: 1.65, Last appended by leader: 65, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } }
I20240501 07:59:51.577258 5700 ts_tablet_manager.cc:1398] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:51.577913 5410 consensus_queue.cc:1035] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 74, Time since last communication: 0.000s
I20240501 07:59:51.579278 3676 raft_consensus.cc:1261] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240501 07:59:51.581308 3699 tablet_copy_service.cc:342] P 014bc50d0c5a44bdbf49c0eb41fa93ad: Request end of tablet copy session da103aa972ab49889fc0bce4abc56bf7-f1aa85993dc449b7b33ebccb4e907046 received from {username='slave'} at 127.0.99.133:43635
I20240501 07:59:51.581405 3699 tablet_copy_service.cc:433] P 014bc50d0c5a44bdbf49c0eb41fa93ad: ending tablet copy session da103aa972ab49889fc0bce4abc56bf7-f1aa85993dc449b7b33ebccb4e907046 on tablet f1aa85993dc449b7b33ebccb4e907046 with peer da103aa972ab49889fc0bce4abc56bf7
W20240501 07:59:51.581863 4623 raft_consensus.cc:1053] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e: Unable to promote non-voter da103aa972ab49889fc0bce4abc56bf7: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: true } }.
Pending config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } }
I20240501 07:59:51.582675 4623 consensus_queue.cc:1035] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 74, Time since last communication: 0.000s
I20240501 07:59:51.583755 3679 raft_consensus.cc:1203] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.74->[1.75-1.76] Dedup: 1.75->[1.76-1.76]
I20240501 07:59:51.587625 5410 raft_consensus.cc:2932] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Committing config change with OpId 1.76: config changed from index 42 to 76, da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.590276 5025 raft_consensus.cc:2932] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index 42 to 76, da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.590487 3529 catalog_manager.cc:5484] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e reported cstate change: config changed from index 42 to 76, da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:51.590700 3679 raft_consensus.cc:2932] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index 42 to 76, da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.592886 5630 tablet_bootstrap.cc:492] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598: Bootstrap replayed 1/1 log segments. Stats: ops{read=57 overwritten=0 applied=56 ignored=0} inserts{seen=728 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:59:51.593272 5630 tablet_bootstrap.cc:492] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598: Bootstrap complete.
I20240501 07:59:51.593461 5630 ts_tablet_manager.cc:1367] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598: Time spent bootstrapping tablet: real 0.517s user 0.073s sys 0.000s
I20240501 07:59:51.594013 5630 raft_consensus.cc:348] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } }
I20240501 07:59:51.596063 5630 raft_consensus.cc:729] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8461c1ff1ed48b09047faeea87e9598, State: Initialized, Role: LEARNER
I20240501 07:59:51.596051 3808 consensus_queue.cc:237] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } }
I20240501 07:59:51.596282 5630 consensus_queue.cc:260] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 56, Last appended: 1.57, Last appended by leader: 57, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } }
I20240501 07:59:51.598147 5630 ts_tablet_manager.cc:1398] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:51.600891 3829 tablet_copy_service.cc:342] P 40c9ccd5f29c424eb845d670a6f7d21e: Request end of tablet copy session d8461c1ff1ed48b09047faeea87e9598-f75acaf04c12462aa4b3fa7e019776d4 received from {username='slave'} at 127.0.99.132:56705
I20240501 07:59:51.600980 3829 tablet_copy_service.cc:433] P 40c9ccd5f29c424eb845d670a6f7d21e: ending tablet copy session d8461c1ff1ed48b09047faeea87e9598-f75acaf04c12462aa4b3fa7e019776d4 on tablet f75acaf04c12462aa4b3fa7e019776d4 with peer d8461c1ff1ed48b09047faeea87e9598
I20240501 07:59:51.601187 5025 raft_consensus.cc:1261] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240501 07:59:51.601569 4623 consensus_queue.cc:1035] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240501 07:59:51.609628 5409 raft_consensus.cc:2932] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index 76 to 77, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.611491 3523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 393b4eef45ff435abd0fde4aec22da88 with cas_config_opid_index 76: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:51.612323 3531 catalog_manager.cc:5484] T 393b4eef45ff435abd0fde4aec22da88 P 40c9ccd5f29c424eb845d670a6f7d21e reported cstate change: config changed from index 76 to 77, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:51.615530 3658 tablet_service.cc:1514] Processing DeleteTablet for tablet 393b4eef45ff435abd0fde4aec22da88 with delete_type TABLET_DATA_TOMBSTONED (TS 014bc50d0c5a44bdbf49c0eb41fa93ad not found in new config with opid_index 77) from {username='slave'} at 127.0.0.1:46686
I20240501 07:59:51.619412 5745 tablet_replica.cc:330] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad: stopping tablet replica
I20240501 07:59:51.625814 5025 raft_consensus.cc:2932] T 393b4eef45ff435abd0fde4aec22da88 P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 76 to 77, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.645257 5745 raft_consensus.cc:2229] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:51.645546 5745 raft_consensus.cc:2258] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:51.648047 4893 raft_consensus.cc:1261] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Refusing update from remote peer 09d8fa5c011b4e398612ba0274c14f2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 07:59:51.649257 5642 consensus_queue.cc:1035] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:51.653551 5745 ts_tablet_manager.cc:1875] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:51.655956 5745 ts_tablet_manager.cc:1888] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.76
I20240501 07:59:51.656670 5745 log.cc:1196] T 393b4eef45ff435abd0fde4aec22da88 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/393b4eef45ff435abd0fde4aec22da88
I20240501 07:59:51.657259 3523 catalog_manager.cc:4830] TS 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453): tablet 393b4eef45ff435abd0fde4aec22da88 (table rebalance_test_table_4 [id=980322bb0877411f904d1f982f67275b]) successfully deleted
I20240501 07:59:51.663281 5025 raft_consensus.cc:1203] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.49->[1.50-1.65] Dedup: 1.60->[1.61-1.65]
I20240501 07:59:51.663375 5681 tablet_bootstrap.cc:492] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=50 overwritten=0 applied=50 ignored=0} inserts{seen=722 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:51.663714 5681 tablet_bootstrap.cc:492] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e: Bootstrap complete.
I20240501 07:59:51.663919 5681 ts_tablet_manager.cc:1367] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e: Time spent bootstrapping tablet: real 0.278s user 0.059s sys 0.002s
I20240501 07:59:51.664399 5681 raft_consensus.cc:348] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } }
I20240501 07:59:51.664623 5681 raft_consensus.cc:729] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09d8fa5c011b4e398612ba0274c14f2e, State: Initialized, Role: LEARNER
I20240501 07:59:51.664801 5681 consensus_queue.cc:260] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 50, Last appended: 1.50, Last appended by leader: 50, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: true } }
I20240501 07:59:51.665241 5681 ts_tablet_manager.cc:1398] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 07:59:51.667174 3961 tablet_copy_service.cc:342] P 8635309522a045b1941fea81699d524c: Request end of tablet copy session 09d8fa5c011b4e398612ba0274c14f2e-54a2b2f834494f29b306e80e48804a43 received from {username='slave'} at 127.0.99.135:60945
I20240501 07:59:51.667266 3961 tablet_copy_service.cc:433] P 8635309522a045b1941fea81699d524c: ending tablet copy session 09d8fa5c011b4e398612ba0274c14f2e-54a2b2f834494f29b306e80e48804a43 on tablet 54a2b2f834494f29b306e80e48804a43 with peer 09d8fa5c011b4e398612ba0274c14f2e
I20240501 07:59:51.679291 5410 raft_consensus.cc:1050] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e: attempting to promote NON_VOTER da103aa972ab49889fc0bce4abc56bf7 to VOTER
I20240501 07:59:51.679674 5410 consensus_queue.cc:237] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 1.66, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } }
I20240501 07:59:51.681529 5424 raft_consensus.cc:1050] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad: attempting to promote NON_VOTER 6208df88dcc34b4daf3d27a574e53a95 to VOTER
I20240501 07:59:51.681876 5424 consensus_queue.cc:237] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 1.68, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } }
I20240501 07:59:51.682502 3678 raft_consensus.cc:1261] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240501 07:59:51.683094 4623 consensus_queue.cc:1035] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20240501 07:59:51.683854 3940 raft_consensus.cc:1261] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240501 07:59:51.683881 5157 raft_consensus.cc:1261] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [term 1 LEARNER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240501 07:59:51.684237 5382 consensus_queue.cc:1035] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 67, Time since last communication: 0.000s
I20240501 07:59:51.684907 5430 consensus_queue.cc:1035] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 67, Time since last communication: 0.000s
I20240501 07:59:51.686934 4623 raft_consensus.cc:2932] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Committing config change with OpId 1.67: config changed from index 50 to 67, da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.688809 5592 raft_consensus.cc:2932] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Committing config change with OpId 1.69: config changed from index 37 to 69, 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:51.689704 3940 raft_consensus.cc:2932] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index 37 to 69, 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:51.690243 5026 raft_consensus.cc:1261] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [term 1 LEARNER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240501 07:59:51.691776 3529 catalog_manager.cc:5484] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: config changed from index 37 to 69, 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" committed_config { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:51.692348 3939 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "839df5100c814d6bb78dc67a8c834a49"
dest_uuid: "8635309522a045b1941fea81699d524c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55596
I20240501 07:59:51.692499 3939 raft_consensus.cc:595] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:51.693470 3531 catalog_manager.cc:5484] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e reported cstate change: config changed from index 50 to 67, da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:51.693764 5371 consensus_queue.cc:1035] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20240501 07:59:51.695389 3679 raft_consensus.cc:2932] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index 50 to 67, da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.696228 5025 raft_consensus.cc:1203] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.51->[1.52-1.70] Dedup: 1.65->[1.66-1.70]
I20240501 07:59:51.699218 5026 raft_consensus.cc:2932] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index 50 to 67, da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.707806 5156 raft_consensus.cc:2932] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index 37 to 69, 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:51.710736 5279 raft_consensus.cc:1203] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.45->[1.46-1.73] Dedup: 1.58->[1.59-1.73]
I20240501 07:59:51.721432 5280 raft_consensus.cc:1203] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.39->[1.40-1.52] Dedup: 1.50->[1.51-1.52]
I20240501 07:59:51.724048 3808 consensus_queue.cc:237] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 1.68, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } }
I20240501 07:59:51.725275 5026 raft_consensus.cc:1261] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240501 07:59:51.726089 5371 consensus_queue.cc:1035] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20240501 07:59:51.728744 5371 raft_consensus.cc:2932] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Committing config change with OpId 1.69: config changed from index 67 to 69, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.729745 5026 raft_consensus.cc:2932] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index 67 to 69, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.732703 5362 raft_consensus.cc:1050] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c: attempting to promote NON_VOTER 09d8fa5c011b4e398612ba0274c14f2e to VOTER
I20240501 07:59:51.733052 5362 consensus_queue.cc:237] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 1.53, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } }
I20240501 07:59:51.733130 3523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5432d22bf6dc4866bbc3d91f21a45c21 with cas_config_opid_index 67: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:51.734617 5280 raft_consensus.cc:1261] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 LEARNER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 53. Preceding OpId from leader: term: 1 index: 54. (index mismatch)
I20240501 07:59:51.735038 4567 consensus_queue.cc:1035] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20240501 07:59:51.737874 3528 catalog_manager.cc:5484] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 reported cstate change: config changed from index 67 to 69, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.743876 3806 raft_consensus.cc:1261] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 53. Preceding OpId from leader: term: 1 index: 54. (index mismatch)
I20240501 07:59:51.743963 3658 tablet_service.cc:1514] Processing DeleteTablet for tablet 5432d22bf6dc4866bbc3d91f21a45c21 with delete_type TABLET_DATA_TOMBSTONED (TS 014bc50d0c5a44bdbf49c0eb41fa93ad not found in new config with opid_index 69) from {username='slave'} at 127.0.0.1:46686
I20240501 07:59:51.744186 5745 tablet_replica.cc:330] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad: stopping tablet replica
I20240501 07:59:51.744320 5745 raft_consensus.cc:2229] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:51.744401 5745 pending_rounds.cc:62] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Trying to abort 1 pending ops.
I20240501 07:59:51.744463 5745 pending_rounds.cc:68] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Aborting op as it isn't in flight: id { term: 1 index: 68 } timestamp: 7022798404474122240 op_type: WRITE_OP write_request { tablet_id: "5432d22bf6dc4866bbc3d91f21a45c21" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000bz\235[\006o\363\346\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""a770ce763e057269" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022798404455288832 authz_token { token_data: "\010\241\357\307\261\006\"3\n\005slave\022*\n 4c0194cc5d4f4b1587cce57551ae4de8\020\001\030\001 \001(\001" signature: "<redacted>""\242\352\370R\373\213\225\n\324\233\030\375\261\237`\267\rRJ\273\013\306\357R\224\376\376\022\225]\221qF\035\034\220@\014.d\234\205\245\320\234\005\017j\017\367\234.rQ5\345W\376\311\216\256\336\214]" signing_key_seq_num: 0 } } request_id { client_id: "8eb27571a07943579f7223ac0b745ac0" seq_no: 184 first_incomplete_seq_no: 184 attempt_no: 0 }
I20240501 07:59:51.744766 5745 raft_consensus.cc:2258] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:51.745488 5745 ts_tablet_manager.cc:1875] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:51.745735 5431 consensus_queue.cc:1035] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20240501 07:59:51.746177 5431 raft_consensus.cc:2932] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Committing config change with OpId 1.54: config changed from index 40 to 54, 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:51.747750 5745 ts_tablet_manager.cc:1888] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.68
I20240501 07:59:51.747814 5745 log.cc:1196] T 5432d22bf6dc4866bbc3d91f21a45c21 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5432d22bf6dc4866bbc3d91f21a45c21
I20240501 07:59:51.748180 5280 raft_consensus.cc:2932] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Committing config change with OpId 1.54: config changed from index 40 to 54, 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:51.748931 3523 catalog_manager.cc:4830] TS 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453): tablet 5432d22bf6dc4866bbc3d91f21a45c21 (table rebalance_test_table_1 [id=4c0194cc5d4f4b1587cce57551ae4de8]) successfully deleted
I20240501 07:59:51.751495 3806 raft_consensus.cc:2932] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Committing config change with OpId 1.54: config changed from index 40 to 54, 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:51.752033 3531 catalog_manager.cc:5484] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c reported cstate change: config changed from index 40 to 54, 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 07:59:51.762820 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240501 07:59:51.765877 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240501 07:59:51.769917 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240501 07:59:51.774080 4893 raft_consensus.cc:1203] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.47] Dedup: 1.40->[1.41-1.47]
I20240501 07:59:51.774966 5155 raft_consensus.cc:1203] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.38->[1.39-1.58] Dedup: 1.50->[1.51-1.58]
W20240501 07:59:51.778663 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240501 07:59:51.779045 3940 consensus_queue.cc:237] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.55, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } }
I20240501 07:59:51.783039 5280 raft_consensus.cc:1261] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240501 07:59:51.783674 5428 consensus_queue.cc:1035] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 54, Time since last communication: 0.000s
W20240501 07:59:51.785382 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240501 07:59:51.788834 5414 raft_consensus.cc:2932] T 54a2b2f834494f29b306e80e48804a43 P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Committing config change with OpId 1.56: config changed from index 54 to 56, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:51.791528 5280 raft_consensus.cc:2932] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index 54 to 56, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:51.793423 3523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 54a2b2f834494f29b306e80e48804a43 with cas_config_opid_index 54: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 07:59:51.793687 3901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240501 07:59:51.808863 3901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240501 07:59:51.814078 5431 raft_consensus.cc:1050] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c: attempting to promote NON_VOTER 09d8fa5c011b4e398612ba0274c14f2e to VOTER
I20240501 07:59:51.814432 5431 consensus_queue.cc:237] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 1.77, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } }
I20240501 07:59:51.815907 5281 raft_consensus.cc:1261] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [term 1 LEARNER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 77. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240501 07:59:51.816268 3678 raft_consensus.cc:1261] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 77. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240501 07:59:51.816815 5431 consensus_queue.cc:1035] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20240501 07:59:51.818822 5429 consensus_queue.cc:1035] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20240501 07:59:51.819612 3528 catalog_manager.cc:5484] T 54a2b2f834494f29b306e80e48804a43 P 09d8fa5c011b4e398612ba0274c14f2e reported cstate change: config changed from index 54 to 56, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:51.822134 3531 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 54a2b2f834494f29b306e80e48804a43 with cas_config_opid_index 54: aborting the task: latest config opid_index 56; task opid_index 54
I20240501 07:59:51.822631 3790 tablet_service.cc:1514] Processing DeleteTablet for tablet 54a2b2f834494f29b306e80e48804a43 with delete_type TABLET_DATA_TOMBSTONED (TS 40c9ccd5f29c424eb845d670a6f7d21e not found in new config with opid_index 56) from {username='slave'} at 127.0.0.1:43228
I20240501 07:59:51.822845 5731 tablet_replica.cc:330] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e: stopping tablet replica
I20240501 07:59:51.822966 5731 raft_consensus.cc:2229] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:51.823076 5731 raft_consensus.cc:2258] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 07:59:51.823112 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240501 07:59:51.823803 5731 ts_tablet_manager.cc:1875] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:51.823863 5428 raft_consensus.cc:2932] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Committing config change with OpId 1.78: config changed from index 46 to 78, 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:51.826862 3679 raft_consensus.cc:2932] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Committing config change with OpId 1.78: config changed from index 46 to 78, 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:51.827950 5731 ts_tablet_manager.cc:1888] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.55
I20240501 07:59:51.828040 5731 log.cc:1196] T 54a2b2f834494f29b306e80e48804a43 P 40c9ccd5f29c424eb845d670a6f7d21e: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/54a2b2f834494f29b306e80e48804a43
I20240501 07:59:51.830873 3523 catalog_manager.cc:4830] TS 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191): tablet 54a2b2f834494f29b306e80e48804a43 (table rebalance_test_table_4 [id=980322bb0877411f904d1f982f67275b]) successfully deleted
I20240501 07:59:51.830987 3528 catalog_manager.cc:5484] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: config changed from index 46 to 78, 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:51.838902 5424 raft_consensus.cc:1050] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad: attempting to promote NON_VOTER da103aa972ab49889fc0bce4abc56bf7 to VOTER
I20240501 07:59:51.839313 5424 consensus_queue.cc:237] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } }
I20240501 07:59:51.840874 5026 raft_consensus.cc:1261] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7 [term 1 LEARNER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240501 07:59:51.841204 5430 consensus_queue.cc:1035] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 70, Time since last communication: 0.000s
I20240501 07:59:51.841878 3808 raft_consensus.cc:1261] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240501 07:59:51.843016 5429 raft_consensus.cc:1050] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c: attempting to promote NON_VOTER d8461c1ff1ed48b09047faeea87e9598 to VOTER
I20240501 07:59:51.843335 5429 consensus_queue.cc:237] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 1.48, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
I20240501 07:59:51.845847 3678 raft_consensus.cc:1261] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 49. (index mismatch)
I20240501 07:59:51.845875 4892 raft_consensus.cc:1261] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [term 1 LEARNER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 49. (index mismatch)
I20240501 07:59:51.846434 5431 consensus_queue.cc:1035] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20240501 07:59:51.848501 5431 consensus_queue.cc:1035] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 47, Time since last communication: 0.000s
I20240501 07:59:51.850214 5428 raft_consensus.cc:2932] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Committing config change with OpId 1.49: config changed from index 28 to 49, d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:51.850737 3679 raft_consensus.cc:2932] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Committing config change with OpId 1.49: config changed from index 28 to 49, d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
W20240501 07:59:51.851536 3901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240501 07:59:51.853742 4893 raft_consensus.cc:2932] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [term 1 FOLLOWER]: Committing config change with OpId 1.49: config changed from index 28 to 49, d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:51.854399 5382 consensus_queue.cc:1035] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 70, Time since last communication: 0.000s
I20240501 07:59:51.857097 5382 raft_consensus.cc:2932] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index 52 to 72, da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.858815 5025 raft_consensus.cc:2932] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 52 to 72, da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.861325 3534 catalog_manager.cc:5484] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7 reported cstate change: config changed from index 52 to 72, da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.864602 3808 raft_consensus.cc:2932] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 52 to 72, da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
W20240501 07:59:51.865888 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240501 07:59:51.868886 5280 raft_consensus.cc:2932] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Committing config change with OpId 1.78: config changed from index 46 to 78, 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:51.869342 3528 catalog_manager.cc:5484] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: config changed from index 28 to 49, d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:51.874614 3679 consensus_queue.cc:237] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.72, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } }
I20240501 07:59:51.876139 5026 raft_consensus.cc:1261] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 73. (index mismatch)
I20240501 07:59:51.876579 5592 consensus_queue.cc:1035] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20240501 07:59:51.879599 5592 raft_consensus.cc:2932] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Committing config change with OpId 1.73: config changed from index 72 to 73, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
I20240501 07:59:51.882040 3523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f1aa85993dc449b7b33ebccb4e907046 with cas_config_opid_index 72: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:51.882656 5026 raft_consensus.cc:2932] T f1aa85993dc449b7b33ebccb4e907046 P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 72 to 73, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } }
W20240501 07:59:51.883687 3901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240501 07:59:51.884459 3531 catalog_manager.cc:5484] T f1aa85993dc449b7b33ebccb4e907046 P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: config changed from index 72 to 73, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" committed_config { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:51.886068 3940 consensus_queue.cc:237] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } }
I20240501 07:59:51.887779 5280 raft_consensus.cc:1261] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240501 07:59:51.888176 5429 consensus_queue.cc:1035] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240501 07:59:51.888952 3790 tablet_service.cc:1514] Processing DeleteTablet for tablet f1aa85993dc449b7b33ebccb4e907046 with delete_type TABLET_DATA_TOMBSTONED (TS 40c9ccd5f29c424eb845d670a6f7d21e not found in new config with opid_index 73) from {username='slave'} at 127.0.0.1:43228
I20240501 07:59:51.889302 5731 tablet_replica.cc:330] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e: stopping tablet replica
I20240501 07:59:51.889436 5731 raft_consensus.cc:2229] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:51.889586 5731 raft_consensus.cc:2258] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:51.890446 5731 ts_tablet_manager.cc:1875] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:51.891002 5429 raft_consensus.cc:2932] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index 78 to 82, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:51.892915 5731 ts_tablet_manager.cc:1888] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.72
I20240501 07:59:51.892987 5731 log.cc:1196] T f1aa85993dc449b7b33ebccb4e907046 P 40c9ccd5f29c424eb845d670a6f7d21e: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f1aa85993dc449b7b33ebccb4e907046
I20240501 07:59:51.893226 3523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cc31e13bb07f44ddb7b80282ee4c434b with cas_config_opid_index 78: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:51.893941 3523 catalog_manager.cc:4830] TS 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191): tablet f1aa85993dc449b7b33ebccb4e907046 (table rebalance_test_table_0 [id=294054b1ea344845ad358beee1ef488a]) successfully deleted
I20240501 07:59:51.893930 5280 raft_consensus.cc:2932] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 78 to 82, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:51.894274 3528 catalog_manager.cc:5484] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c reported cstate change: config changed from index 78 to 82, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 07:59:51.895823 3901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240501 07:59:51.897969 3658 tablet_service.cc:1514] Processing DeleteTablet for tablet cc31e13bb07f44ddb7b80282ee4c434b with delete_type TABLET_DATA_TOMBSTONED (TS 014bc50d0c5a44bdbf49c0eb41fa93ad not found in new config with opid_index 82) from {username='slave'} at 127.0.0.1:46686
I20240501 07:59:51.899127 4893 raft_consensus.cc:1203] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.47->[1.48-1.74] Dedup: 1.57->[1.58-1.74]
I20240501 07:59:51.899494 5745 tablet_replica.cc:330] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad: stopping tablet replica
I20240501 07:59:51.899639 5745 raft_consensus.cc:2229] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:51.899770 5745 raft_consensus.cc:2258] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:51.900702 5745 ts_tablet_manager.cc:1875] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:51.903117 5745 ts_tablet_manager.cc:1888] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.81
I20240501 07:59:51.903201 5745 log.cc:1196] T cc31e13bb07f44ddb7b80282ee4c434b P 014bc50d0c5a44bdbf49c0eb41fa93ad: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/cc31e13bb07f44ddb7b80282ee4c434b
I20240501 07:59:51.903844 3523 catalog_manager.cc:4830] TS 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453): tablet cc31e13bb07f44ddb7b80282ee4c434b (table rebalance_test_table_2 [id=712e7be85ad54a6ebf02a160dca63546]) successfully deleted
W20240501 07:59:51.915354 3901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240501 07:59:51.924721 4623 raft_consensus.cc:1050] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e: attempting to promote NON_VOTER d8461c1ff1ed48b09047faeea87e9598 to VOTER
I20240501 07:59:51.925021 4623 consensus_queue.cc:237] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 1.75, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
I20240501 07:59:51.926645 3939 raft_consensus.cc:1261] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240501 07:59:51.927450 4892 raft_consensus.cc:1261] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [term 1 LEARNER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240501 07:59:51.928258 5409 consensus_queue.cc:1035] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240501 07:59:51.928936 5409 consensus_queue.cc:1035] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 74, Time since last communication: 0.000s
W20240501 07:59:51.930795 3901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240501 07:59:51.932641 5371 raft_consensus.cc:2932] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Committing config change with OpId 1.76: config changed from index 48 to 76, d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:51.932927 4892 raft_consensus.cc:2932] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index 48 to 76, d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:51.933032 3940 raft_consensus.cc:2932] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index 48 to 76, d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:51.935843 3534 catalog_manager.cc:5484] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 reported cstate change: config changed from index 48 to 76, d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
W20240501 07:59:51.947782 3900 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240501 07:59:51.966403 3900 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240501 07:59:51.989130 3900 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240501 07:59:52.007532 3900 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240501 07:59:52.027628 3900 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240501 07:59:52.031481 3940 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31a0bc2519cd43eda2f2fc8b5a6e7d5b"
dest_uuid: "8635309522a045b1941fea81699d524c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55640
I20240501 07:59:52.031695 3940 raft_consensus.cc:595] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:52.037604 5431 raft_consensus.cc:979] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c: : Instructing follower d8461c1ff1ed48b09047faeea87e9598 to start an election
I20240501 07:59:52.037722 5431 raft_consensus.cc:1067] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Signalling peer d8461c1ff1ed48b09047faeea87e9598 to start an election
I20240501 07:59:52.038730 4892 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "31a0bc2519cd43eda2f2fc8b5a6e7d5b"
dest_uuid: "d8461c1ff1ed48b09047faeea87e9598"
from {username='slave'} at 127.0.99.131:33417
I20240501 07:59:52.038914 4892 raft_consensus.cc:482] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:52.038991 4892 raft_consensus.cc:3037] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:52.040086 4892 raft_consensus.cc:504] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
I20240501 07:59:52.040609 4892 leader_election.cc:288] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [CANDIDATE]: Term 2 election: Requested vote from peers 8635309522a045b1941fea81699d524c (127.0.99.131:39655), 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:52.042223 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a0bc2519cd43eda2f2fc8b5a6e7d5b" candidate_uuid: "d8461c1ff1ed48b09047faeea87e9598" candidate_term: 2 candidate_status { last_received { term: 1 index: 57 } } ignore_live_leader: true dest_uuid: "8635309522a045b1941fea81699d524c"
I20240501 07:59:52.042364 3940 raft_consensus.cc:3032] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:52.042407 3940 raft_consensus.cc:729] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8635309522a045b1941fea81699d524c, State: Running, Role: LEADER
I20240501 07:59:52.042551 3940 consensus_queue.cc:260] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.57, Last appended by leader: 57, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
I20240501 07:59:52.042721 3940 raft_consensus.cc:3037] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:52.043944 3940 raft_consensus.cc:2454] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8461c1ff1ed48b09047faeea87e9598 in term 2.
I20240501 07:59:52.044266 4827 leader_election.cc:302] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8635309522a045b1941fea81699d524c, d8461c1ff1ed48b09047faeea87e9598; no voters:
I20240501 07:59:52.044801 5753 raft_consensus.cc:2781] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:52.045147 5753 raft_consensus.cc:686] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [term 2 LEADER]: Becoming Leader. State: Replica: d8461c1ff1ed48b09047faeea87e9598, State: Running, Role: LEADER
I20240501 07:59:52.045518 5753 consensus_queue.cc:237] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.57, Last appended by leader: 57, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
I20240501 07:59:52.048524 3531 catalog_manager.cc:5484] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 reported cstate change: term changed from 1 to 2, leader changed from 8635309522a045b1941fea81699d524c (127.0.99.131) to d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "d8461c1ff1ed48b09047faeea87e9598" committed_config { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:52.049937 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a0bc2519cd43eda2f2fc8b5a6e7d5b" candidate_uuid: "d8461c1ff1ed48b09047faeea87e9598" candidate_term: 2 candidate_status { last_received { term: 1 index: 57 } } ignore_live_leader: true dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
I20240501 07:59:52.050149 3678 raft_consensus.cc:3037] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:52.050434 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240501 07:59:52.051071 3678 raft_consensus.cc:2454] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8461c1ff1ed48b09047faeea87e9598 in term 2.
I20240501 07:59:52.070420 3679 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "02f63ed2bd6345b686c74eb002bb306f"
dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44960
I20240501 07:59:52.070611 3679 raft_consensus.cc:595] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Received request to transfer leadership
W20240501 07:59:52.072587 3900 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240501 07:59:52.081634 3900 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55456: Illegal state: replica 8635309522a045b1941fea81699d524c is not leader of this config: current role FOLLOWER
W20240501 07:59:52.096305 3638 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44976: Illegal state: replica 014bc50d0c5a44bdbf49c0eb41fa93ad is not leader of this config: current role FOLLOWER
W20240501 07:59:52.096603 3900 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240501 07:59:52.099066 3808 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f75acaf04c12462aa4b3fa7e019776d4"
dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34746
I20240501 07:59:52.099244 3808 raft_consensus.cc:595] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Received request to transfer leadership
W20240501 07:59:52.103304 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
W20240501 07:59:52.106959 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
W20240501 07:59:52.112402 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.115975 3679 raft_consensus.cc:1261] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Refusing update from remote peer d8461c1ff1ed48b09047faeea87e9598: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 2 index: 59. (index mismatch)
I20240501 07:59:52.116011 3940 raft_consensus.cc:1261] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Refusing update from remote peer d8461c1ff1ed48b09047faeea87e9598: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 2 index: 59. (index mismatch)
I20240501 07:59:52.116461 5729 consensus_queue.cc:1035] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20240501 07:59:52.116650 5753 consensus_queue.cc:1035] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 56, Time since last communication: 0.000s
W20240501 07:59:52.119899 3900 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240501 07:59:52.121251 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.122335 5821 mvcc.cc:205] Tried to move back new op lower bound from 7022798406102581248 to 7022798405821378560. Current Snapshot: MvccSnapshot[applied={T|T < 7022798406102581248}]
I20240501 07:59:52.130822 4892 consensus_queue.cc:237] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 2.59, Last appended by leader: 57, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
W20240501 07:59:52.132373 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.134676 3679 raft_consensus.cc:1261] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Refusing update from remote peer d8461c1ff1ed48b09047faeea87e9598: Log matching property violated. Preceding OpId in replica: term: 2 index: 59. Preceding OpId from leader: term: 2 index: 60. (index mismatch)
I20240501 07:59:52.135066 5863 consensus_queue.cc:1035] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20240501 07:59:52.136847 5753 raft_consensus.cc:2932] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [term 2 LEADER]: Committing config change with OpId 2.60: config changed from index 49 to 60, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:52.137254 3679 raft_consensus.cc:2932] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Committing config change with OpId 2.60: config changed from index 49 to 60, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:52.139884 3523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b with cas_config_opid_index 49: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:52.140007 3531 catalog_manager.cc:5484] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: config changed from index 49 to 60, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "d8461c1ff1ed48b09047faeea87e9598" committed_config { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
W20240501 07:59:52.141458 3770 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.142980 5428 raft_consensus.cc:979] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c: : Instructing follower 014bc50d0c5a44bdbf49c0eb41fa93ad to start an election
I20240501 07:59:52.143090 5428 raft_consensus.cc:1067] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Signalling peer 014bc50d0c5a44bdbf49c0eb41fa93ad to start an election
I20240501 07:59:52.143558 3679 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "839df5100c814d6bb78dc67a8c834a49"
dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
from {username='slave'} at 127.0.99.131:57645
I20240501 07:59:52.143699 3679 raft_consensus.cc:482] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:52.143765 3679 raft_consensus.cc:3037] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:52.146179 3920 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b with delete_type TABLET_DATA_TOMBSTONED (TS 8635309522a045b1941fea81699d524c not found in new config with opid_index 60) from {username='slave'} at 127.0.0.1:53920
I20240501 07:59:52.144781 3679 raft_consensus.cc:504] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } }
I20240501 07:59:52.148367 3939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "839df5100c814d6bb78dc67a8c834a49" candidate_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 56 } } ignore_live_leader: true dest_uuid: "8635309522a045b1941fea81699d524c"
I20240501 07:59:52.148504 3939 raft_consensus.cc:3032] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:52.148557 3939 raft_consensus.cc:729] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8635309522a045b1941fea81699d524c, State: Running, Role: LEADER
I20240501 07:59:52.148718 3939 consensus_queue.cc:260] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 1.56, Last appended by leader: 56, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } }
I20240501 07:59:52.148890 3939 raft_consensus.cc:3037] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:52.149160 3679 leader_election.cc:288] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 2 election: Requested vote from peers 8635309522a045b1941fea81699d524c (127.0.99.131:39655), 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135:41559)
I20240501 07:59:52.150091 3939 raft_consensus.cc:2454] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014bc50d0c5a44bdbf49c0eb41fa93ad in term 2.
I20240501 07:59:52.150394 3614 leader_election.cc:302] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014bc50d0c5a44bdbf49c0eb41fa93ad, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:52.150576 5430 raft_consensus.cc:2781] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:52.150676 5430 raft_consensus.cc:686] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 LEADER]: Becoming Leader. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Running, Role: LEADER
W20240501 07:59:52.150758 3900 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Illegal state: replica 8635309522a045b1941fea81699d524c is not leader of this config: current role FOLLOWER
I20240501 07:59:52.150811 5430 consensus_queue.cc:237] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 1.56, Last appended by leader: 56, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } }
I20240501 07:59:52.151247 3939 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "839df5100c814d6bb78dc67a8c834a49"
dest_uuid: "8635309522a045b1941fea81699d524c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55654
I20240501 07:59:52.151351 3939 raft_consensus.cc:595] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Received request to transfer leadership
I20240501 07:59:52.151399 3939 raft_consensus.cc:603] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240501 07:59:52.152662 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.152752 3534 catalog_manager.cc:5484] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: term changed from 1 to 2, leader changed from 8635309522a045b1941fea81699d524c (127.0.99.131) to 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129). New cstate: current_term: 2 leader_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" committed_config { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:52.165524 5865 tablet_replica.cc:330] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c: stopping tablet replica
W20240501 07:59:52.165585 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.165691 5865 raft_consensus.cc:2229] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:52.165825 5865 raft_consensus.cc:2258] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:52.169245 5280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "839df5100c814d6bb78dc67a8c834a49" candidate_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 56 } } ignore_live_leader: true dest_uuid: "09d8fa5c011b4e398612ba0274c14f2e"
I20240501 07:59:52.169371 5280 raft_consensus.cc:3037] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:52.170518 5280 raft_consensus.cc:2454] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014bc50d0c5a44bdbf49c0eb41fa93ad in term 2.
W20240501 07:59:52.175516 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.179540 5279 raft_consensus.cc:1261] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e [term 2 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 2 index: 58. (index mismatch)
I20240501 07:59:52.179704 3939 raft_consensus.cc:1261] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 2 index: 58. (index mismatch)
I20240501 07:59:52.180073 5430 consensus_queue.cc:1035] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20240501 07:59:52.180404 5430 consensus_queue.cc:1035] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20240501 07:59:52.184577 5405 mvcc.cc:205] Tried to move back new op lower bound from 7022798406363770880 to 7022798406251163648. Current Snapshot: MvccSnapshot[applied={T|T < 7022798406363770880 or (T in {7022798406363770880})}]
I20240501 07:59:52.186002 5794 mvcc.cc:205] Tried to move back new op lower bound from 7022798406363770880 to 7022798406251163648. Current Snapshot: MvccSnapshot[applied={T|T < 7022798406363770880 or (T in {7022798406363770880})}]
W20240501 07:59:52.188838 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.191958 5414 raft_consensus.cc:1050] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c: attempting to promote NON_VOTER 6208df88dcc34b4daf3d27a574e53a95 to VOTER
I20240501 07:59:52.192193 3679 consensus_queue.cc:237] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 2.58, Last appended by leader: 56, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } }
I20240501 07:59:52.192365 5414 consensus_queue.cc:237] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 1.59, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } }
I20240501 07:59:52.193661 5279 raft_consensus.cc:1261] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e [term 2 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 58. Preceding OpId from leader: term: 2 index: 59. (index mismatch)
I20240501 07:59:52.193854 5157 raft_consensus.cc:1261] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95 [term 1 LEARNER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch)
I20240501 07:59:52.194058 5424 consensus_queue.cc:1035] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20240501 07:59:52.194286 5428 consensus_queue.cc:1035] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 58, Time since last communication: 0.000s
I20240501 07:59:52.195731 3808 raft_consensus.cc:1261] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch)
I20240501 07:59:52.196131 5429 raft_consensus.cc:2932] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Committing config change with OpId 1.60: config changed from index 39 to 60, 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:52.196491 5424 raft_consensus.cc:2932] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 LEADER]: Committing config change with OpId 2.59: config changed from index 48 to 59, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:52.198457 3523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 839df5100c814d6bb78dc67a8c834a49 with cas_config_opid_index 48: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:52.198689 5431 consensus_queue.cc:1035] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 58, Time since last communication: 0.000s
I20240501 07:59:52.198982 5280 raft_consensus.cc:2932] T 839df5100c814d6bb78dc67a8c834a49 P 09d8fa5c011b4e398612ba0274c14f2e [term 2 FOLLOWER]: Committing config change with OpId 2.59: config changed from index 48 to 59, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } }
I20240501 07:59:52.200595 3531 catalog_manager.cc:5484] T 839df5100c814d6bb78dc67a8c834a49 P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: config changed from index 48 to 59, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:52.202237 5157 raft_consensus.cc:2932] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95 [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index 39 to 60, 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
W20240501 07:59:52.204119 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.204046 3808 raft_consensus.cc:2932] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index 39 to 60, 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:52.204361 3920 tablet_service.cc:1514] Processing DeleteTablet for tablet 839df5100c814d6bb78dc67a8c834a49 with delete_type TABLET_DATA_TOMBSTONED (TS 8635309522a045b1941fea81699d524c not found in new config with opid_index 59) from {username='slave'} at 127.0.0.1:53920
I20240501 07:59:52.206467 5865 ts_tablet_manager.cc:1875] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:52.207298 3534 catalog_manager.cc:5484] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95 reported cstate change: config changed from index 39 to 60, 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:52.210338 5865 ts_tablet_manager.cc:1888] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.59
I20240501 07:59:52.212911 3939 consensus_queue.cc:237] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.61, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } }
I20240501 07:59:52.214087 5865 log.cc:1196] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 8635309522a045b1941fea81699d524c: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/31a0bc2519cd43eda2f2fc8b5a6e7d5b
I20240501 07:59:52.214639 5865 tablet_replica.cc:330] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c: stopping tablet replica
I20240501 07:59:52.216044 5155 raft_consensus.cc:1261] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95 [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240501 07:59:52.216305 5865 raft_consensus.cc:2229] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:52.216495 5865 raft_consensus.cc:2258] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:52.216953 5414 consensus_queue.cc:1035] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 60, Time since last communication: 0.000s
I20240501 07:59:52.217406 5865 ts_tablet_manager.cc:1875] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:52.217810 3523 catalog_manager.cc:4830] TS 8635309522a045b1941fea81699d524c (127.0.99.131:39655): tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b (table rebalance_test_table_4 [id=980322bb0877411f904d1f982f67275b]) successfully deleted
W20240501 07:59:52.218729 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.219976 5431 raft_consensus.cc:2932] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Committing config change with OpId 1.62: config changed from index 60 to 62, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:52.220680 5865 ts_tablet_manager.cc:1888] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.58
I20240501 07:59:52.220752 5865 log.cc:1196] T 839df5100c814d6bb78dc67a8c834a49 P 8635309522a045b1941fea81699d524c: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/839df5100c814d6bb78dc67a8c834a49
I20240501 07:59:52.222688 3523 catalog_manager.cc:4830] TS 8635309522a045b1941fea81699d524c (127.0.99.131:39655): tablet 839df5100c814d6bb78dc67a8c834a49 (table rebalance_test_table_3 [id=86dc286087384ca29a85396def16cf86]) successfully deleted
I20240501 07:59:52.222844 3523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c02a65bc31814809ae316ae5fdd07a6e with cas_config_opid_index 60: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:52.223104 3528 catalog_manager.cc:5484] T c02a65bc31814809ae316ae5fdd07a6e P 8635309522a045b1941fea81699d524c reported cstate change: config changed from index 60 to 62, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:52.223366 5155 raft_consensus.cc:2932] T c02a65bc31814809ae316ae5fdd07a6e P 6208df88dcc34b4daf3d27a574e53a95 [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index 60 to 62, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:52.226091 3790 tablet_service.cc:1514] Processing DeleteTablet for tablet c02a65bc31814809ae316ae5fdd07a6e with delete_type TABLET_DATA_TOMBSTONED (TS 40c9ccd5f29c424eb845d670a6f7d21e not found in new config with opid_index 62) from {username='slave'} at 127.0.0.1:43228
I20240501 07:59:52.226298 5731 tablet_replica.cc:330] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e: stopping tablet replica
I20240501 07:59:52.226418 5731 raft_consensus.cc:2229] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:52.226505 5731 pending_rounds.cc:62] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e: Trying to abort 1 pending ops.
I20240501 07:59:52.226567 5731 pending_rounds.cc:68] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e: Aborting op as it isn't in flight: id { term: 1 index: 61 } timestamp: 7022798406464397312 op_type: WRITE_OP write_request { tablet_id: "c02a65bc31814809ae316ae5fdd07a6e" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\250$W!\310C*\312\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""cc0b4448ef5ff3f1" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022798406402170880 authz_token { token_data: "\010\241\357\307\261\006\"3\n\005slave\022*\n 86dc286087384ca29a85396def16cf86\020\001\030\001 \001(\001" signature: "<redacted>""hC@\352\220\302\311\2544\212\013\260@\034W\331\324\276\206\257xr\357x<\302\256\270\234o\026\357.Y\257\250\306\017-\347\2140\2608<1p\226\367\216AD\252\367m\346\216|; \0337\246\016" signing_key_seq_num: 0 } } request_id { client_id: "6a8eac1fd714441babb4457419b703e8" seq_no: 157 first_incomplete_seq_no: 157 attempt_no: 0 }
I20240501 07:59:52.226876 5731 raft_consensus.cc:2258] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:52.227671 5731 ts_tablet_manager.cc:1875] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:52.230094 5731 ts_tablet_manager.cc:1888] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.61
I20240501 07:59:52.230175 5731 log.cc:1196] T c02a65bc31814809ae316ae5fdd07a6e P 40c9ccd5f29c424eb845d670a6f7d21e: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c02a65bc31814809ae316ae5fdd07a6e
I20240501 07:59:52.231549 3523 catalog_manager.cc:4830] TS 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191): tablet c02a65bc31814809ae316ae5fdd07a6e (table rebalance_test_table_3 [id=86dc286087384ca29a85396def16cf86]) successfully deleted
W20240501 07:59:52.236747 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.241791 3679 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "02f63ed2bd6345b686c74eb002bb306f"
dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45000
I20240501 07:59:52.241950 3679 raft_consensus.cc:595] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Received request to transfer leadership
W20240501 07:59:52.251935 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
W20240501 07:59:52.267809 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.271493 3808 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f75acaf04c12462aa4b3fa7e019776d4"
dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34774
I20240501 07:59:52.271631 3808 raft_consensus.cc:595] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Received request to transfer leadership
W20240501 07:59:52.288717 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
W20240501 07:59:52.307384 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.328236 3679 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "02f63ed2bd6345b686c74eb002bb306f"
dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45016
W20240501 07:59:52.328341 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.328405 3679 raft_consensus.cc:595] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:52.332607 5424 raft_consensus.cc:979] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad: : Instructing follower 6208df88dcc34b4daf3d27a574e53a95 to start an election
I20240501 07:59:52.332716 5424 raft_consensus.cc:1067] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Signalling peer 6208df88dcc34b4daf3d27a574e53a95 to start an election
I20240501 07:59:52.333323 5155 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "02f63ed2bd6345b686c74eb002bb306f"
dest_uuid: "6208df88dcc34b4daf3d27a574e53a95"
from {username='slave'} at 127.0.99.129:38943
I20240501 07:59:52.333519 5155 raft_consensus.cc:482] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:52.333626 5155 raft_consensus.cc:3037] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:52.334774 5155 raft_consensus.cc:504] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } }
I20240501 07:59:52.335950 5155 leader_election.cc:288] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [CANDIDATE]: Term 2 election: Requested vote from peers 8635309522a045b1941fea81699d524c (127.0.99.131:39655), 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:52.336249 3939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f63ed2bd6345b686c74eb002bb306f" candidate_uuid: "6208df88dcc34b4daf3d27a574e53a95" candidate_term: 2 candidate_status { last_received { term: 1 index: 76 } } ignore_live_leader: true dest_uuid: "8635309522a045b1941fea81699d524c"
I20240501 07:59:52.336371 3939 raft_consensus.cc:3037] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:52.336382 3679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f63ed2bd6345b686c74eb002bb306f" candidate_uuid: "6208df88dcc34b4daf3d27a574e53a95" candidate_term: 2 candidate_status { last_received { term: 1 index: 76 } } ignore_live_leader: true dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
I20240501 07:59:52.336486 3679 raft_consensus.cc:3032] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:52.336526 3679 raft_consensus.cc:729] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Running, Role: LEADER
I20240501 07:59:52.336686 3679 consensus_queue.cc:260] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 76, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } }
I20240501 07:59:52.336848 3679 raft_consensus.cc:3037] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:52.337435 3939 raft_consensus.cc:2454] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6208df88dcc34b4daf3d27a574e53a95 in term 2.
I20240501 07:59:52.337885 3679 raft_consensus.cc:2454] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6208df88dcc34b4daf3d27a574e53a95 in term 2.
I20240501 07:59:52.337906 5092 leader_election.cc:302] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6208df88dcc34b4daf3d27a574e53a95, 8635309522a045b1941fea81699d524c; no voters:
I20240501 07:59:52.338224 5658 raft_consensus.cc:2781] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:52.338424 5658 raft_consensus.cc:686] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [term 2 LEADER]: Becoming Leader. State: Replica: 6208df88dcc34b4daf3d27a574e53a95, State: Running, Role: LEADER
I20240501 07:59:52.338733 5658 consensus_queue.cc:237] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 76, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } }
I20240501 07:59:52.341815 3531 catalog_manager.cc:5484] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 reported cstate change: term changed from 1 to 2, leader changed from 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) to 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "6208df88dcc34b4daf3d27a574e53a95" committed_config { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 07:59:52.351840 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.367667 3808 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f75acaf04c12462aa4b3fa7e019776d4"
dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34782
I20240501 07:59:52.367846 3808 raft_consensus.cc:595] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Received request to transfer leadership
W20240501 07:59:52.374706 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
W20240501 07:59:52.400424 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
W20240501 07:59:52.424499 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.437733 3806 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f75acaf04c12462aa4b3fa7e019776d4"
dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34792
I20240501 07:59:52.437899 3806 raft_consensus.cc:595] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Received request to transfer leadership
W20240501 07:59:52.451965 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
W20240501 07:59:52.479014 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
W20240501 07:59:52.506770 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
W20240501 07:59:52.523761 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44848: Illegal state: replica 014bc50d0c5a44bdbf49c0eb41fa93ad is not leader of this config: current role FOLLOWER
W20240501 07:59:52.528054 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55414: Illegal state: replica 8635309522a045b1941fea81699d524c is not leader of this config: current role FOLLOWER
W20240501 07:59:52.539650 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Service unavailable: leader transfer in progress
I20240501 07:59:52.544917 5410 raft_consensus.cc:979] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e: : Instructing follower d8461c1ff1ed48b09047faeea87e9598 to start an election
I20240501 07:59:52.545084 5410 raft_consensus.cc:1067] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Signalling peer d8461c1ff1ed48b09047faeea87e9598 to start an election
I20240501 07:59:52.545668 4892 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f75acaf04c12462aa4b3fa7e019776d4"
dest_uuid: "d8461c1ff1ed48b09047faeea87e9598"
from {username='slave'} at 127.0.99.130:44129
I20240501 07:59:52.545830 4892 raft_consensus.cc:482] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:52.546039 4892 raft_consensus.cc:3037] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:52.547166 4892 raft_consensus.cc:504] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
I20240501 07:59:52.547639 4892 leader_election.cc:288] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [CANDIDATE]: Term 2 election: Requested vote from peers 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191), 8635309522a045b1941fea81699d524c (127.0.99.131:39655)
I20240501 07:59:52.548177 3939 raft_consensus.cc:1261] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Refusing update from remote peer 6208df88dcc34b4daf3d27a574e53a95: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 2 index: 78. (index mismatch)
I20240501 07:59:52.548650 3679 raft_consensus.cc:1261] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Refusing update from remote peer 6208df88dcc34b4daf3d27a574e53a95: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 2 index: 78. (index mismatch)
I20240501 07:59:52.548827 5658 consensus_queue.cc:1035] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240501 07:59:52.548930 3939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75acaf04c12462aa4b3fa7e019776d4" candidate_uuid: "d8461c1ff1ed48b09047faeea87e9598" candidate_term: 2 candidate_status { last_received { term: 1 index: 85 } } ignore_live_leader: true dest_uuid: "8635309522a045b1941fea81699d524c"
I20240501 07:59:52.549059 3939 raft_consensus.cc:3037] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:52.549499 5658 consensus_queue.cc:1035] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240501 07:59:52.550243 3939 raft_consensus.cc:2454] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8461c1ff1ed48b09047faeea87e9598 in term 2.
I20240501 07:59:52.550550 4827 leader_election.cc:302] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8635309522a045b1941fea81699d524c, d8461c1ff1ed48b09047faeea87e9598; no voters:
I20240501 07:59:52.550742 5863 raft_consensus.cc:2781] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:52.550851 5863 raft_consensus.cc:686] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [term 2 LEADER]: Becoming Leader. State: Replica: d8461c1ff1ed48b09047faeea87e9598, State: Running, Role: LEADER
I20240501 07:59:52.550995 5863 consensus_queue.cc:237] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 1.85, Last appended by leader: 85, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
I20240501 07:59:52.553737 3531 catalog_manager.cc:5484] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 reported cstate change: term changed from 1 to 2, leader changed from 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) to d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "d8461c1ff1ed48b09047faeea87e9598" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:52.555692 3806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75acaf04c12462aa4b3fa7e019776d4" candidate_uuid: "d8461c1ff1ed48b09047faeea87e9598" candidate_term: 2 candidate_status { last_received { term: 1 index: 85 } } ignore_live_leader: true dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
I20240501 07:59:52.555853 3806 raft_consensus.cc:3032] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:52.555912 3806 raft_consensus.cc:729] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Running, Role: LEADER
I20240501 07:59:52.556097 3806 consensus_queue.cc:260] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 1.85, Last appended by leader: 85, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
I20240501 07:59:52.556308 3806 raft_consensus.cc:3037] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:52.557425 3806 raft_consensus.cc:2454] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8461c1ff1ed48b09047faeea87e9598 in term 2.
I20240501 07:59:52.569309 5157 consensus_queue.cc:237] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 2.78, Last appended by leader: 76, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } }
W20240501 07:59:52.569615 3769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34544: Illegal state: replica 40c9ccd5f29c424eb845d670a6f7d21e is not leader of this config: current role FOLLOWER
I20240501 07:59:52.571025 3940 raft_consensus.cc:1261] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Refusing update from remote peer 6208df88dcc34b4daf3d27a574e53a95: Log matching property violated. Preceding OpId in replica: term: 2 index: 78. Preceding OpId from leader: term: 2 index: 79. (index mismatch)
I20240501 07:59:52.572711 5954 consensus_queue.cc:1035] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240501 07:59:52.575080 5658 raft_consensus.cc:2932] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [term 2 LEADER]: Committing config change with OpId 2.79: config changed from index 69 to 79, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:52.576814 3524 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 02f63ed2bd6345b686c74eb002bb306f with cas_config_opid_index 69: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:52.576813 3940 raft_consensus.cc:2932] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Committing config change with OpId 2.79: config changed from index 69 to 79, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:52.577716 3531 catalog_manager.cc:5484] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 reported cstate change: config changed from index 69 to 79, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "6208df88dcc34b4daf3d27a574e53a95" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:52.583353 3658 tablet_service.cc:1514] Processing DeleteTablet for tablet 02f63ed2bd6345b686c74eb002bb306f with delete_type TABLET_DATA_TOMBSTONED (TS 014bc50d0c5a44bdbf49c0eb41fa93ad not found in new config with opid_index 79) from {username='slave'} at 127.0.0.1:46686
I20240501 07:59:52.583909 5955 tablet_replica.cc:330] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad: stopping tablet replica
I20240501 07:59:52.584118 5955 raft_consensus.cc:2229] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:52.584283 5955 raft_consensus.cc:2258] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:52.585649 5955 ts_tablet_manager.cc:1875] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:52.591441 5955 ts_tablet_manager.cc:1888] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.78
I20240501 07:59:52.591539 5955 log.cc:1196] T 02f63ed2bd6345b686c74eb002bb306f P 014bc50d0c5a44bdbf49c0eb41fa93ad: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/02f63ed2bd6345b686c74eb002bb306f
I20240501 07:59:52.592100 3523 catalog_manager.cc:4830] TS 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453): tablet 02f63ed2bd6345b686c74eb002bb306f (table rebalance_test_table_0 [id=294054b1ea344845ad358beee1ef488a]) successfully deleted
W20240501 07:59:52.602205 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55430: Illegal state: replica 8635309522a045b1941fea81699d524c is not leader of this config: current role FOLLOWER
I20240501 07:59:52.650629 3808 raft_consensus.cc:1261] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 2 FOLLOWER]: Refusing update from remote peer d8461c1ff1ed48b09047faeea87e9598: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 2 index: 87. (index mismatch)
I20240501 07:59:52.650908 3940 raft_consensus.cc:1261] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Refusing update from remote peer d8461c1ff1ed48b09047faeea87e9598: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 2 index: 87. (index mismatch)
I20240501 07:59:52.651432 5863 consensus_queue.cc:1035] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20240501 07:59:52.651741 5863 consensus_queue.cc:1035] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20240501 07:59:52.662748 4892 consensus_queue.cc:237] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 2.87, Last appended by leader: 85, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
I20240501 07:59:52.666536 3939 raft_consensus.cc:1261] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Refusing update from remote peer d8461c1ff1ed48b09047faeea87e9598: Log matching property violated. Preceding OpId in replica: term: 2 index: 87. Preceding OpId from leader: term: 2 index: 88. (index mismatch)
I20240501 07:59:52.668485 5729 consensus_queue.cc:1035] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20240501 07:59:52.671103 5729 raft_consensus.cc:2932] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [term 2 LEADER]: Committing config change with OpId 2.88: config changed from index 76 to 88, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:52.673102 3523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f75acaf04c12462aa4b3fa7e019776d4 with cas_config_opid_index 76: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:52.674058 3531 catalog_manager.cc:5484] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 reported cstate change: config changed from index 76 to 88, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "d8461c1ff1ed48b09047faeea87e9598" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:52.674952 3939 raft_consensus.cc:2932] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Committing config change with OpId 2.88: config changed from index 76 to 88, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:52.682619 3790 tablet_service.cc:1514] Processing DeleteTablet for tablet f75acaf04c12462aa4b3fa7e019776d4 with delete_type TABLET_DATA_TOMBSTONED (TS 40c9ccd5f29c424eb845d670a6f7d21e not found in new config with opid_index 88) from {username='slave'} at 127.0.0.1:43228
I20240501 07:59:52.682854 5731 tablet_replica.cc:330] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e: stopping tablet replica
I20240501 07:59:52.682981 5731 raft_consensus.cc:2229] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:52.683096 5731 raft_consensus.cc:2258] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:52.683949 5731 ts_tablet_manager.cc:1875] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:52.686671 5731 ts_tablet_manager.cc:1888] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.87
I20240501 07:59:52.686753 5731 log.cc:1196] T f75acaf04c12462aa4b3fa7e019776d4 P 40c9ccd5f29c424eb845d670a6f7d21e: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f75acaf04c12462aa4b3fa7e019776d4
I20240501 07:59:52.687355 3523 catalog_manager.cc:4830] TS 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191): tablet f75acaf04c12462aa4b3fa7e019776d4 (table rebalance_test_table_1 [id=4c0194cc5d4f4b1587cce57551ae4de8]) successfully deleted
I20240501 07:59:52.846846 5268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:52.861332 5006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:52.865796 5137 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:52.881032 3790 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:52.881709 3658 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:52.886682 4871 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:52.906299 3920 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:52.998605 5157 consensus_queue.cc:237] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 2.91, Last appended by leader: 76, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } }
I20240501 07:59:53.000623 3940 raft_consensus.cc:1261] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Refusing update from remote peer 6208df88dcc34b4daf3d27a574e53a95: Log matching property violated. Preceding OpId in replica: term: 2 index: 91. Preceding OpId from leader: term: 2 index: 92. (index mismatch)
I20240501 07:59:53.000982 5928 consensus_queue.cc:1035] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20240501 07:59:53.002825 5928 raft_consensus.cc:2932] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [term 2 LEADER]: Committing config change with OpId 2.92: config changed from index 79 to 92, NON_VOTER d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) added. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } } }
I20240501 07:59:53.006213 3529 catalog_manager.cc:5484] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 reported cstate change: config changed from index 79 to 92, NON_VOTER d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) added. New cstate: current_term: 2 leader_uuid: "6208df88dcc34b4daf3d27a574e53a95" committed_config { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:53.009887 3940 raft_consensus.cc:2932] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Committing config change with OpId 2.92: config changed from index 79 to 92, NON_VOTER d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) added. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } } }
W20240501 07:59:53.029789 5092 consensus_peers.cc:485] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 -> Peer d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132:37117): Couldn't send request to peer d8461c1ff1ed48b09047faeea87e9598. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02f63ed2bd6345b686c74eb002bb306f. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:53.037710 3679 consensus_queue.cc:237] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } }
I20240501 07:59:53.042930 3806 raft_consensus.cc:1261] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240501 07:59:53.043368 5592 consensus_queue.cc:1035] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240501 07:59:53.046731 4668 raft_consensus.cc:2932] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index -1 to 77, NON_VOTER d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) added. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } } }
I20240501 07:59:53.047328 3806 raft_consensus.cc:2932] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index -1 to 77, NON_VOTER d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) added. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } } }
I20240501 07:59:53.049639 3529 catalog_manager.cc:5484] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: config changed from index -1 to 77, NON_VOTER d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:53.050462 3614 consensus_peers.cc:485] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad -> Peer d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132:37117): Couldn't send request to peer d8461c1ff1ed48b09047faeea87e9598. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b23c9dc72fbc4e409f1d22506b11cb42. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:53.114588 4892 consensus_queue.cc:237] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 2.103, Last appended by leader: 85, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } }
I20240501 07:59:53.116160 3940 raft_consensus.cc:1261] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Refusing update from remote peer d8461c1ff1ed48b09047faeea87e9598: Log matching property violated. Preceding OpId in replica: term: 2 index: 103. Preceding OpId from leader: term: 2 index: 104. (index mismatch)
I20240501 07:59:53.116566 5863 consensus_queue.cc:1035] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240501 07:59:53.118175 5863 raft_consensus.cc:2932] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [term 2 LEADER]: Committing config change with OpId 2.104: config changed from index 88 to 104, NON_VOTER 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) added. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } } }
I20240501 07:59:53.121920 3534 catalog_manager.cc:5484] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 reported cstate change: config changed from index 88 to 104, NON_VOTER 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) added. New cstate: current_term: 2 leader_uuid: "d8461c1ff1ed48b09047faeea87e9598" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:53.126173 4828 consensus_peers.cc:485] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 -> Peer 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134:36011): Couldn't send request to peer 6208df88dcc34b4daf3d27a574e53a95. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f75acaf04c12462aa4b3fa7e019776d4. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:53.126304 3939 raft_consensus.cc:2932] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Committing config change with OpId 2.104: config changed from index 88 to 104, NON_VOTER 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) added. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } } }
I20240501 07:59:53.396802 6056 ts_tablet_manager.cc:902] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598: Initiating tablet copy from peer 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134:36011)
I20240501 07:59:53.403677 6056 tablet_copy_client.cc:308] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:36011
I20240501 07:59:53.412547 5177 tablet_copy_service.cc:140] P 6208df88dcc34b4daf3d27a574e53a95: Received BeginTabletCopySession request for tablet 02f63ed2bd6345b686c74eb002bb306f from peer d8461c1ff1ed48b09047faeea87e9598 ({username='slave'} at 127.0.99.132:35333)
I20240501 07:59:53.412721 5177 tablet_copy_service.cc:161] P 6208df88dcc34b4daf3d27a574e53a95: Beginning new tablet copy session on tablet 02f63ed2bd6345b686c74eb002bb306f from peer d8461c1ff1ed48b09047faeea87e9598 at {username='slave'} at 127.0.99.132:35333: session id = d8461c1ff1ed48b09047faeea87e9598-02f63ed2bd6345b686c74eb002bb306f
I20240501 07:59:53.414691 5177 tablet_copy_source_session.cc:205] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:53.419608 6056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02f63ed2bd6345b686c74eb002bb306f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:53.422330 6056 tablet_copy_client.cc:783] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:53.422729 6056 tablet_copy_client.cc:647] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:53.425089 6056 tablet_copy_client.cc:523] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:53.427947 6056 tablet_bootstrap.cc:492] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598: Bootstrap starting.
I20240501 07:59:53.553937 6074 ts_tablet_manager.cc:902] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598: Initiating tablet copy from peer 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453)
I20240501 07:59:53.554739 6074 tablet_copy_client.cc:308] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42453
I20240501 07:59:53.555312 3699 tablet_copy_service.cc:140] P 014bc50d0c5a44bdbf49c0eb41fa93ad: Received BeginTabletCopySession request for tablet b23c9dc72fbc4e409f1d22506b11cb42 from peer d8461c1ff1ed48b09047faeea87e9598 ({username='slave'} at 127.0.99.132:60837)
I20240501 07:59:53.555440 3699 tablet_copy_service.cc:161] P 014bc50d0c5a44bdbf49c0eb41fa93ad: Beginning new tablet copy session on tablet b23c9dc72fbc4e409f1d22506b11cb42 from peer d8461c1ff1ed48b09047faeea87e9598 at {username='slave'} at 127.0.99.132:60837: session id = d8461c1ff1ed48b09047faeea87e9598-b23c9dc72fbc4e409f1d22506b11cb42
I20240501 07:59:53.556416 3699 tablet_copy_source_session.cc:205] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:53.562141 6074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23c9dc72fbc4e409f1d22506b11cb42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:53.566332 6074 tablet_copy_client.cc:783] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:53.566541 6074 tablet_copy_client.cc:647] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:53.569805 6074 tablet_copy_client.cc:523] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:53.570209 6079 ts_tablet_manager.cc:902] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95: Initiating tablet copy from peer d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132:37117)
I20240501 07:59:53.570854 6079 tablet_copy_client.cc:308] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37117
I20240501 07:59:53.572234 4913 tablet_copy_service.cc:140] P d8461c1ff1ed48b09047faeea87e9598: Received BeginTabletCopySession request for tablet f75acaf04c12462aa4b3fa7e019776d4 from peer 6208df88dcc34b4daf3d27a574e53a95 ({username='slave'} at 127.0.99.134:41367)
I20240501 07:59:53.572388 4913 tablet_copy_service.cc:161] P d8461c1ff1ed48b09047faeea87e9598: Beginning new tablet copy session on tablet f75acaf04c12462aa4b3fa7e019776d4 from peer 6208df88dcc34b4daf3d27a574e53a95 at {username='slave'} at 127.0.99.134:41367: session id = 6208df88dcc34b4daf3d27a574e53a95-f75acaf04c12462aa4b3fa7e019776d4
I20240501 07:59:53.573724 4913 tablet_copy_source_session.cc:205] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:53.575491 6079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75acaf04c12462aa4b3fa7e019776d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:53.577913 6079 tablet_copy_client.cc:783] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:53.579007 6079 tablet_copy_client.cc:647] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:53.582264 6074 tablet_bootstrap.cc:492] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598: Bootstrap starting.
I20240501 07:59:53.630311 6056 tablet_bootstrap.cc:492] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598: Bootstrap replayed 1/1 log segments. Stats: ops{read=105 overwritten=0 applied=105 ignored=0} inserts{seen=732 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:53.630769 6056 tablet_bootstrap.cc:492] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598: Bootstrap complete.
I20240501 07:59:53.631073 6056 ts_tablet_manager.cc:1367] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598: Time spent bootstrapping tablet: real 0.203s user 0.095s sys 0.007s
I20240501 07:59:53.631686 6056 raft_consensus.cc:348] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } }
I20240501 07:59:53.631912 6056 raft_consensus.cc:729] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d8461c1ff1ed48b09047faeea87e9598, State: Initialized, Role: LEARNER
I20240501 07:59:53.632071 6056 consensus_queue.cc:260] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 105, Last appended: 2.105, Last appended by leader: 105, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } }
I20240501 07:59:53.633141 6056 ts_tablet_manager.cc:1398] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:53.633839 5177 tablet_copy_service.cc:342] P 6208df88dcc34b4daf3d27a574e53a95: Request end of tablet copy session d8461c1ff1ed48b09047faeea87e9598-02f63ed2bd6345b686c74eb002bb306f received from {username='slave'} at 127.0.99.132:35333
I20240501 07:59:53.633937 5177 tablet_copy_service.cc:433] P 6208df88dcc34b4daf3d27a574e53a95: ending tablet copy session d8461c1ff1ed48b09047faeea87e9598-02f63ed2bd6345b686c74eb002bb306f on tablet 02f63ed2bd6345b686c74eb002bb306f with peer d8461c1ff1ed48b09047faeea87e9598
I20240501 07:59:53.659230 6079 tablet_copy_client.cc:523] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:53.660862 6079 tablet_bootstrap.cc:492] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95: Bootstrap starting.
I20240501 07:59:53.834251 6074 tablet_bootstrap.cc:492] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598: Bootstrap replayed 1/1 log segments. Stats: ops{read=92 overwritten=0 applied=92 ignored=0} inserts{seen=777 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:53.834767 6074 tablet_bootstrap.cc:492] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598: Bootstrap complete.
I20240501 07:59:53.835053 6074 ts_tablet_manager.cc:1367] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598: Time spent bootstrapping tablet: real 0.253s user 0.085s sys 0.010s
I20240501 07:59:53.835860 6074 raft_consensus.cc:348] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } }
I20240501 07:59:53.836133 6074 raft_consensus.cc:729] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8461c1ff1ed48b09047faeea87e9598, State: Initialized, Role: LEARNER
I20240501 07:59:53.836304 6074 consensus_queue.cc:260] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 92, Last appended: 1.92, Last appended by leader: 92, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: true } }
I20240501 07:59:53.836752 6074 ts_tablet_manager.cc:1398] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:53.842288 3699 tablet_copy_service.cc:342] P 014bc50d0c5a44bdbf49c0eb41fa93ad: Request end of tablet copy session d8461c1ff1ed48b09047faeea87e9598-b23c9dc72fbc4e409f1d22506b11cb42 received from {username='slave'} at 127.0.99.132:60837
I20240501 07:59:53.842388 3699 tablet_copy_service.cc:433] P 014bc50d0c5a44bdbf49c0eb41fa93ad: ending tablet copy session d8461c1ff1ed48b09047faeea87e9598-b23c9dc72fbc4e409f1d22506b11cb42 on tablet b23c9dc72fbc4e409f1d22506b11cb42 with peer d8461c1ff1ed48b09047faeea87e9598
I20240501 07:59:53.904002 6079 tablet_bootstrap.cc:492] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95: Bootstrap replayed 1/1 log segments. Stats: ops{read=122 overwritten=0 applied=121 ignored=0} inserts{seen=789 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:59:53.904640 6079 tablet_bootstrap.cc:492] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95: Bootstrap complete.
I20240501 07:59:53.905957 6079 ts_tablet_manager.cc:1367] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95: Time spent bootstrapping tablet: real 0.245s user 0.102s sys 0.011s
I20240501 07:59:53.906558 6079 raft_consensus.cc:348] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95 [term 2 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } }
I20240501 07:59:53.906975 6079 raft_consensus.cc:729] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6208df88dcc34b4daf3d27a574e53a95, State: Initialized, Role: LEARNER
I20240501 07:59:53.907465 6079 consensus_queue.cc:260] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 121, Last appended: 2.122, Last appended by leader: 122, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: true } }
I20240501 07:59:53.908057 6079 ts_tablet_manager.cc:1398] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:53.910712 4913 tablet_copy_service.cc:342] P d8461c1ff1ed48b09047faeea87e9598: Request end of tablet copy session 6208df88dcc34b4daf3d27a574e53a95-f75acaf04c12462aa4b3fa7e019776d4 received from {username='slave'} at 127.0.99.134:41367
I20240501 07:59:53.910817 4913 tablet_copy_service.cc:433] P d8461c1ff1ed48b09047faeea87e9598: ending tablet copy session 6208df88dcc34b4daf3d27a574e53a95-f75acaf04c12462aa4b3fa7e019776d4 on tablet f75acaf04c12462aa4b3fa7e019776d4 with peer 6208df88dcc34b4daf3d27a574e53a95
I20240501 07:59:53.984968 4892 raft_consensus.cc:1203] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.76->[1.77-1.103] Dedup: 1.92->[1.93-1.103]
I20240501 07:59:54.003320 4892 raft_consensus.cc:1203] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.91->[2.92-2.125] Dedup: 2.105->[2.106-2.125]
I20240501 07:59:54.013613 5156 raft_consensus.cc:1203] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.103->[2.104-2.132] Dedup: 2.122->[2.123-2.132]
I20240501 07:59:54.049503 5863 raft_consensus.cc:1050] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598: attempting to promote NON_VOTER 6208df88dcc34b4daf3d27a574e53a95 to VOTER
I20240501 07:59:54.049928 5863 consensus_queue.cc:237] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 132, Committed index: 132, Last appended: 2.133, Last appended by leader: 85, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } }
I20240501 07:59:54.051960 5154 raft_consensus.cc:1261] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95 [term 2 LEARNER]: Refusing update from remote peer d8461c1ff1ed48b09047faeea87e9598: Log matching property violated. Preceding OpId in replica: term: 2 index: 133. Preceding OpId from leader: term: 2 index: 134. (index mismatch)
I20240501 07:59:54.052313 6051 consensus_queue.cc:1035] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 132, Time since last communication: 0.000s
I20240501 07:59:54.052541 3940 raft_consensus.cc:1261] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Refusing update from remote peer d8461c1ff1ed48b09047faeea87e9598: Log matching property violated. Preceding OpId in replica: term: 2 index: 133. Preceding OpId from leader: term: 2 index: 134. (index mismatch)
I20240501 07:59:54.053058 6051 consensus_queue.cc:1035] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 132, Time since last communication: 0.000s
I20240501 07:59:54.054383 6051 raft_consensus.cc:2932] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [term 2 LEADER]: Committing config change with OpId 2.134: config changed from index 104 to 134, 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:54.056880 5154 raft_consensus.cc:2932] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95 [term 2 FOLLOWER]: Committing config change with OpId 2.134: config changed from index 104 to 134, 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:54.057116 3534 catalog_manager.cc:5484] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 reported cstate change: config changed from index 104 to 134, 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d8461c1ff1ed48b09047faeea87e9598" committed_config { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:54.058521 5928 raft_consensus.cc:1050] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95: attempting to promote NON_VOTER d8461c1ff1ed48b09047faeea87e9598 to VOTER
I20240501 07:59:54.058967 5928 consensus_queue.cc:237] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 2.127, Last appended by leader: 76, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
I20240501 07:59:54.062168 4893 raft_consensus.cc:1261] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598 [term 2 LEARNER]: Refusing update from remote peer 6208df88dcc34b4daf3d27a574e53a95: Log matching property violated. Preceding OpId in replica: term: 2 index: 127. Preceding OpId from leader: term: 2 index: 128. (index mismatch)
I20240501 07:59:54.062404 3940 raft_consensus.cc:2932] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Committing config change with OpId 2.134: config changed from index 104 to 134, 6208df88dcc34b4daf3d27a574e53a95 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:54.063599 5382 raft_consensus.cc:1050] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad: attempting to promote NON_VOTER d8461c1ff1ed48b09047faeea87e9598 to VOTER
I20240501 07:59:54.063891 5382 consensus_queue.cc:237] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
I20240501 07:59:54.064931 5954 consensus_queue.cc:1035] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 126, Time since last communication: 0.000s
I20240501 07:59:54.067209 3808 raft_consensus.cc:1261] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240501 07:59:54.067750 5430 consensus_queue.cc:1035] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240501 07:59:54.068969 4893 raft_consensus.cc:1261] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598 [term 1 LEARNER]: Refusing update from remote peer 014bc50d0c5a44bdbf49c0eb41fa93ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240501 07:59:54.070690 5430 consensus_queue.cc:1035] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240501 07:59:54.070870 5954 raft_consensus.cc:2932] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [term 2 LEADER]: Committing config change with OpId 2.128: config changed from index 92 to 128, d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:54.073854 4892 raft_consensus.cc:2932] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Committing config change with OpId 2.128: config changed from index 92 to 128, d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:54.074083 5382 raft_consensus.cc:2932] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index 77 to 105, d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:54.076365 4893 raft_consensus.cc:2932] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 77 to 105, d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:54.076661 3528 catalog_manager.cc:5484] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: config changed from index 77 to 105, d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:54.077344 3939 raft_consensus.cc:1261] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Refusing update from remote peer 6208df88dcc34b4daf3d27a574e53a95: Log matching property violated. Preceding OpId in replica: term: 2 index: 126. Preceding OpId from leader: term: 2 index: 128. (index mismatch)
I20240501 07:59:54.080077 3808 raft_consensus.cc:2932] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 77 to 105, d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:54.081758 5928 consensus_queue.cc:1035] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 125, Time since last communication: 0.000s
I20240501 07:59:54.091980 4892 consensus_queue.cc:237] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 135, Committed index: 135, Last appended: 2.135, Last appended by leader: 85, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } }
I20240501 07:59:54.092674 3531 catalog_manager.cc:5484] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 reported cstate change: config changed from index 92 to 128, d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6208df88dcc34b4daf3d27a574e53a95" committed_config { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:54.095207 5154 raft_consensus.cc:1261] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95 [term 2 FOLLOWER]: Refusing update from remote peer d8461c1ff1ed48b09047faeea87e9598: Log matching property violated. Preceding OpId in replica: term: 2 index: 135. Preceding OpId from leader: term: 2 index: 137. (index mismatch)
I20240501 07:59:54.096537 5968 consensus_queue.cc:1035] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 135, Time since last communication: 0.000s
I20240501 07:59:54.098101 3940 raft_consensus.cc:2932] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Committing config change with OpId 2.128: config changed from index 92 to 128, d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:54.098623 5968 raft_consensus.cc:2932] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 [term 2 LEADER]: Committing config change with OpId 2.136: config changed from index 134 to 136, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New config: { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:54.099401 5154 consensus_queue.cc:237] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 2.129, Last appended by leader: 76, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
I20240501 07:59:54.100167 3523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f75acaf04c12462aa4b3fa7e019776d4 with cas_config_opid_index 134: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:54.102895 3536 catalog_manager.cc:5484] T f75acaf04c12462aa4b3fa7e019776d4 P d8461c1ff1ed48b09047faeea87e9598 reported cstate change: config changed from index 134 to 136, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "d8461c1ff1ed48b09047faeea87e9598" committed_config { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:54.103013 5154 raft_consensus.cc:2932] T f75acaf04c12462aa4b3fa7e019776d4 P 6208df88dcc34b4daf3d27a574e53a95 [term 2 FOLLOWER]: Committing config change with OpId 2.136: config changed from index 134 to 136, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New config: { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } }
I20240501 07:59:54.103334 4892 raft_consensus.cc:1261] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Refusing update from remote peer 6208df88dcc34b4daf3d27a574e53a95: Log matching property violated. Preceding OpId in replica: term: 2 index: 129. Preceding OpId from leader: term: 2 index: 130. (index mismatch)
I20240501 07:59:54.106665 5928 consensus_queue.cc:1035] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 128, Time since last communication: 0.000s
I20240501 07:59:54.107014 3920 tablet_service.cc:1514] Processing DeleteTablet for tablet f75acaf04c12462aa4b3fa7e019776d4 with delete_type TABLET_DATA_TOMBSTONED (TS 8635309522a045b1941fea81699d524c not found in new config with opid_index 136) from {username='slave'} at 127.0.0.1:53920
I20240501 07:59:54.107399 6125 tablet_replica.cc:330] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c: stopping tablet replica
I20240501 07:59:54.107620 6125 raft_consensus.cc:2229] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:54.107851 6125 raft_consensus.cc:2258] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:54.109097 6125 ts_tablet_manager.cc:1875] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:54.111020 5658 raft_consensus.cc:2932] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 [term 2 LEADER]: Committing config change with OpId 2.130: config changed from index 128 to 130, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:54.112684 3524 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 02f63ed2bd6345b686c74eb002bb306f with cas_config_opid_index 128: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:54.113749 4892 raft_consensus.cc:2932] T 02f63ed2bd6345b686c74eb002bb306f P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Committing config change with OpId 2.130: config changed from index 128 to 130, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:54.114840 3536 catalog_manager.cc:5484] T 02f63ed2bd6345b686c74eb002bb306f P 6208df88dcc34b4daf3d27a574e53a95 reported cstate change: config changed from index 128 to 130, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "6208df88dcc34b4daf3d27a574e53a95" committed_config { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "6208df88dcc34b4daf3d27a574e53a95" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:54.119277 3920 tablet_service.cc:1514] Processing DeleteTablet for tablet 02f63ed2bd6345b686c74eb002bb306f with delete_type TABLET_DATA_TOMBSTONED (TS 8635309522a045b1941fea81699d524c not found in new config with opid_index 130) from {username='slave'} at 127.0.0.1:53920
I20240501 07:59:54.146332 6125 ts_tablet_manager.cc:1888] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.135
I20240501 07:59:54.146448 6125 log.cc:1196] T f75acaf04c12462aa4b3fa7e019776d4 P 8635309522a045b1941fea81699d524c: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f75acaf04c12462aa4b3fa7e019776d4
I20240501 07:59:54.147027 6125 tablet_replica.cc:330] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c: stopping tablet replica
I20240501 07:59:54.147089 3523 catalog_manager.cc:4830] TS 8635309522a045b1941fea81699d524c (127.0.99.131:39655): tablet f75acaf04c12462aa4b3fa7e019776d4 (table rebalance_test_table_1 [id=4c0194cc5d4f4b1587cce57551ae4de8]) successfully deleted
I20240501 07:59:54.147500 6125 raft_consensus.cc:2229] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:54.148739 6125 raft_consensus.cc:2258] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:54.149612 6125 ts_tablet_manager.cc:1875] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:54.152592 6125 ts_tablet_manager.cc:1888] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.128
I20240501 07:59:54.152732 6125 log.cc:1196] T 02f63ed2bd6345b686c74eb002bb306f P 8635309522a045b1941fea81699d524c: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/02f63ed2bd6345b686c74eb002bb306f
I20240501 07:59:54.153787 3523 catalog_manager.cc:4830] TS 8635309522a045b1941fea81699d524c (127.0.99.131:39655): tablet 02f63ed2bd6345b686c74eb002bb306f (table rebalance_test_table_0 [id=294054b1ea344845ad358beee1ef488a]) successfully deleted
I20240501 07:59:54.240088 3679 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b23c9dc72fbc4e409f1d22506b11cb42"
dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45088
I20240501 07:59:54.240257 3679 raft_consensus.cc:595] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:54.332445 3679 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b23c9dc72fbc4e409f1d22506b11cb42"
dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45104
I20240501 07:59:54.332592 3679 raft_consensus.cc:595] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Received request to transfer leadership
W20240501 07:59:54.332681 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.341828 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.349432 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.359251 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.366762 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.379129 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.397099 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.410717 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.427742 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.443250 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.458500 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.475510 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.492918 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.507822 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.526762 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.548715 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.571198 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
I20240501 07:59:54.584152 3679 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b23c9dc72fbc4e409f1d22506b11cb42"
dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45120
I20240501 07:59:54.584328 3679 raft_consensus.cc:595] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Received request to transfer leadership
W20240501 07:59:54.592902 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.613037 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.636351 3638 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
W20240501 07:59:54.667932 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Service unavailable: leader transfer in progress
I20240501 07:59:54.670192 6122 raft_consensus.cc:979] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad: : Instructing follower 40c9ccd5f29c424eb845d670a6f7d21e to start an election
I20240501 07:59:54.670310 6122 raft_consensus.cc:1067] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Signalling peer 40c9ccd5f29c424eb845d670a6f7d21e to start an election
I20240501 07:59:54.674285 3808 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b23c9dc72fbc4e409f1d22506b11cb42"
dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
from {username='slave'} at 127.0.99.129:56403
I20240501 07:59:54.674464 3808 raft_consensus.cc:482] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:54.674532 3808 raft_consensus.cc:3037] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:54.675729 3808 raft_consensus.cc:504] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
I20240501 07:59:54.676122 3808 leader_election.cc:288] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 2 election: Requested vote from peers 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453), d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132:37117)
I20240501 07:59:54.677795 4893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23c9dc72fbc4e409f1d22506b11cb42" candidate_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" candidate_term: 2 candidate_status { last_received { term: 1 index: 111 } } ignore_live_leader: true dest_uuid: "d8461c1ff1ed48b09047faeea87e9598"
I20240501 07:59:54.677937 4893 raft_consensus.cc:3037] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:54.678174 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23c9dc72fbc4e409f1d22506b11cb42" candidate_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" candidate_term: 2 candidate_status { last_received { term: 1 index: 111 } } ignore_live_leader: true dest_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad"
I20240501 07:59:54.678282 3678 raft_consensus.cc:3032] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:54.678320 3678 raft_consensus.cc:729] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 014bc50d0c5a44bdbf49c0eb41fa93ad, State: Running, Role: LEADER
I20240501 07:59:54.678458 3678 consensus_queue.cc:260] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 1.111, Last appended by leader: 111, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
I20240501 07:59:54.678618 3678 raft_consensus.cc:3037] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:54.679163 4893 raft_consensus.cc:2454] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c9ccd5f29c424eb845d670a6f7d21e in term 2.
I20240501 07:59:54.679513 3745 leader_election.cc:302] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c9ccd5f29c424eb845d670a6f7d21e, d8461c1ff1ed48b09047faeea87e9598; no voters:
I20240501 07:59:54.679657 3678 raft_consensus.cc:2454] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c9ccd5f29c424eb845d670a6f7d21e in term 2.
I20240501 07:59:54.680230 4623 raft_consensus.cc:2781] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:54.680336 4623 raft_consensus.cc:686] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 2 LEADER]: Becoming Leader. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Running, Role: LEADER
I20240501 07:59:54.680485 4623 consensus_queue.cc:237] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 1.111, Last appended by leader: 111, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
I20240501 07:59:54.682484 3536 catalog_manager.cc:5484] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e reported cstate change: term changed from 1 to 2, leader changed from 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) to 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130). New cstate: current_term: 2 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:54.695976 3639 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44862: Illegal state: replica 014bc50d0c5a44bdbf49c0eb41fa93ad is not leader of this config: current role FOLLOWER
I20240501 07:59:54.745627 3678 raft_consensus.cc:1261] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 2 index: 113. (index mismatch)
I20240501 07:59:54.745630 4893 raft_consensus.cc:1261] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 2 index: 113. (index mismatch)
I20240501 07:59:54.749087 5410 consensus_queue.cc:1035] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20240501 07:59:54.749464 5410 consensus_queue.cc:1035] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20240501 07:59:54.757679 3806 consensus_queue.cc:237] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 2.113, Last appended by leader: 111, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } }
I20240501 07:59:54.765597 4893 raft_consensus.cc:1261] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 2 index: 113. Preceding OpId from leader: term: 2 index: 114. (index mismatch)
I20240501 07:59:54.766652 5409 consensus_queue.cc:1035] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20240501 07:59:54.772049 4623 raft_consensus.cc:2932] T b23c9dc72fbc4e409f1d22506b11cb42 P 40c9ccd5f29c424eb845d670a6f7d21e [term 2 LEADER]: Committing config change with OpId 2.114: config changed from index 105 to 114, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:54.772650 4893 raft_consensus.cc:2932] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Committing config change with OpId 2.114: config changed from index 105 to 114, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:54.774102 3523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b23c9dc72fbc4e409f1d22506b11cb42 with cas_config_opid_index 105: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:54.776859 3528 catalog_manager.cc:5484] T b23c9dc72fbc4e409f1d22506b11cb42 P d8461c1ff1ed48b09047faeea87e9598 reported cstate change: config changed from index 105 to 114, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } }
I20240501 07:59:54.781236 3658 tablet_service.cc:1514] Processing DeleteTablet for tablet b23c9dc72fbc4e409f1d22506b11cb42 with delete_type TABLET_DATA_TOMBSTONED (TS 014bc50d0c5a44bdbf49c0eb41fa93ad not found in new config with opid_index 114) from {username='slave'} at 127.0.0.1:46686
I20240501 07:59:54.786191 6178 tablet_replica.cc:330] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad: stopping tablet replica
I20240501 07:59:54.786412 6178 raft_consensus.cc:2229] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:54.786582 6178 raft_consensus.cc:2258] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:54.788014 6178 ts_tablet_manager.cc:1875] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:54.799165 6178 ts_tablet_manager.cc:1888] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.113
I20240501 07:59:54.799386 6178 log.cc:1196] T b23c9dc72fbc4e409f1d22506b11cb42 P 014bc50d0c5a44bdbf49c0eb41fa93ad: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b23c9dc72fbc4e409f1d22506b11cb42
I20240501 07:59:54.801443 3523 catalog_manager.cc:4830] TS 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453): tablet b23c9dc72fbc4e409f1d22506b11cb42 (table rebalance_test_table_3 [id=86dc286087384ca29a85396def16cf86]) successfully deleted
I20240501 07:59:55.077551 5137 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:55.082590 3790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:55.083078 3658 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:55.087396 3920 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:55.090950 5268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:55.102427 5006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.111510 4871 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:55.563566 5137 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:55.566686 3658 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:55.568965 5268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:55.570355 3920 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:55.610198 5006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.646675 3790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:55.648012 4871 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:56.427477 5137 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:56.431174 3658 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:56.440743 4871 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:56.461266 3920 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:56.496896 5268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:56.514673 5006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:56.554466 3790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:56.861868 5268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:56.864389 5137 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:56.872539 3658 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:56.884364 4871 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:56.885453 3790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:56.911950 5006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:56.938568 3920 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:57.106822 3658 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:57.111759 3790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:57.113359 5268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:57.118229 3920 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:57.120401 5137 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:57.121289 4871 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:57.135217 5006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
2033c6e9bc084ce3baa7be66008fcca6 | 127.0.99.190:44911 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:39583 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
014bc50d0c5a44bdbf49c0eb41fa93ad | 127.0.99.129:42453 | HEALTHY | <none> | 3 | 0
09d8fa5c011b4e398612ba0274c14f2e | 127.0.99.135:41559 | HEALTHY | <none> | 1 | 0
40c9ccd5f29c424eb845d670a6f7d21e | 127.0.99.130:43191 | HEALTHY | <none> | 4 | 0
6208df88dcc34b4daf3d27a574e53a95 | 127.0.99.134:36011 | HEALTHY | <none> | 1 | 0
8635309522a045b1941fea81699d524c | 127.0.99.131:39655 | HEALTHY | <none> | 5 | 0
d8461c1ff1ed48b09047faeea87e9598 | 127.0.99.132:37117 | HEALTHY | <none> | 2 | 0
da103aa972ab49889fc0bce4abc56bf7 | 127.0.99.133:40225 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:42453
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:43191
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:39655
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:37117
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:40225
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:36011
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:41559
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:42453
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:43191
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:39655
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:37117
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:40225
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:36011
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:41559
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:39583 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 07:59:57.177495 398 log_verifier.cc:126] Checking tablet 02f63ed2bd6345b686c74eb002bb306f
I20240501 07:59:57.234675 398 log_verifier.cc:177] Verified matching terms for 238 ops in tablet 02f63ed2bd6345b686c74eb002bb306f
I20240501 07:59:57.234871 398 log_verifier.cc:126] Checking tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b
I20240501 07:59:57.292508 398 log_verifier.cc:177] Verified matching terms for 228 ops in tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b
I20240501 07:59:57.292630 398 log_verifier.cc:126] Checking tablet 393b4eef45ff435abd0fde4aec22da88
I20240501 07:59:57.348574 398 log_verifier.cc:177] Verified matching terms for 265 ops in tablet 393b4eef45ff435abd0fde4aec22da88
I20240501 07:59:57.348783 398 log_verifier.cc:126] Checking tablet 5432d22bf6dc4866bbc3d91f21a45c21
I20240501 07:59:57.400336 398 log_verifier.cc:177] Verified matching terms for 231 ops in tablet 5432d22bf6dc4866bbc3d91f21a45c21
I20240501 07:59:57.400446 398 log_verifier.cc:126] Checking tablet 54a2b2f834494f29b306e80e48804a43
I20240501 07:59:57.447448 398 log_verifier.cc:177] Verified matching terms for 223 ops in tablet 54a2b2f834494f29b306e80e48804a43
I20240501 07:59:57.447582 398 log_verifier.cc:126] Checking tablet 71a2915d4efd4d48a1d41df5ecf2f4d7
I20240501 07:59:57.462145 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 71a2915d4efd4d48a1d41df5ecf2f4d7
I20240501 07:59:57.462211 398 log_verifier.cc:126] Checking tablet 72c5108aa1aa4cbd913d9ceb1a262d58
I20240501 07:59:57.510084 398 log_verifier.cc:177] Verified matching terms for 225 ops in tablet 72c5108aa1aa4cbd913d9ceb1a262d58
I20240501 07:59:57.510188 398 log_verifier.cc:126] Checking tablet 7c0cda2d32534bb29a6a2538ac13368a
I20240501 07:59:57.563159 398 log_verifier.cc:177] Verified matching terms for 263 ops in tablet 7c0cda2d32534bb29a6a2538ac13368a
I20240501 07:59:57.563274 398 log_verifier.cc:126] Checking tablet 839df5100c814d6bb78dc67a8c834a49
I20240501 07:59:57.604198 398 log_verifier.cc:177] Verified matching terms for 183 ops in tablet 839df5100c814d6bb78dc67a8c834a49
I20240501 07:59:57.604308 398 log_verifier.cc:126] Checking tablet a77d5e3c352f495cbee97be6a2c4661a
I20240501 07:59:57.646888 398 log_verifier.cc:177] Verified matching terms for 198 ops in tablet a77d5e3c352f495cbee97be6a2c4661a
I20240501 07:59:57.646998 398 log_verifier.cc:126] Checking tablet b23c9dc72fbc4e409f1d22506b11cb42
I20240501 07:59:57.691329 398 log_verifier.cc:177] Verified matching terms for 199 ops in tablet b23c9dc72fbc4e409f1d22506b11cb42
I20240501 07:59:57.691453 398 log_verifier.cc:126] Checking tablet c02a65bc31814809ae316ae5fdd07a6e
I20240501 07:59:57.739676 398 log_verifier.cc:177] Verified matching terms for 218 ops in tablet c02a65bc31814809ae316ae5fdd07a6e
I20240501 07:59:57.739789 398 log_verifier.cc:126] Checking tablet cc31e13bb07f44ddb7b80282ee4c434b
I20240501 07:59:57.795827 398 log_verifier.cc:177] Verified matching terms for 282 ops in tablet cc31e13bb07f44ddb7b80282ee4c434b
I20240501 07:59:57.795961 398 log_verifier.cc:126] Checking tablet ed30e32a4ba2483484b5bc5e3cfb44fd
I20240501 07:59:57.847780 398 log_verifier.cc:177] Verified matching terms for 253 ops in tablet ed30e32a4ba2483484b5bc5e3cfb44fd
I20240501 07:59:57.847919 398 log_verifier.cc:126] Checking tablet f1aa85993dc449b7b33ebccb4e907046
I20240501 07:59:57.896353 398 log_verifier.cc:177] Verified matching terms for 229 ops in tablet f1aa85993dc449b7b33ebccb4e907046
I20240501 07:59:57.896485 398 log_verifier.cc:126] Checking tablet f75acaf04c12462aa4b3fa7e019776d4
I20240501 07:59:57.943995 398 log_verifier.cc:177] Verified matching terms for 234 ops in tablet f75acaf04c12462aa4b3fa7e019776d4
I20240501 07:59:57.946385 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44911
--builtin_ntp_servers=127.0.99.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 07:59:58.069928 6403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:58.070209 6403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:58.070271 6403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:58.073460 6403 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:58.073535 6403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:58.073624 6403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240501 07:59:58.077368 6403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44911
--never_fsync=true
--heap_profile_path=/tmp/kudu.6403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:58.078315 6403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:58.079505 6403 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 07:59:58.089082 6403 server_base.cc:1034] running on GCE node
W20240501 07:59:58.089769 6410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:58.090891 6409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:58.091715 6412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:58.092171 6403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:58.092770 6403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:58.093916 6403 hybrid_clock.cc:648] HybridClock initialized: now 1714550398093896 us; error 35 us; skew 500 ppm
I20240501 07:59:58.096038 6403 webserver.cc:460] Webserver started at http://127.0.99.136:42915/ using document root <none> and password file <none>
I20240501 07:59:58.096635 6403 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:58.096693 6403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:58.096874 6403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:58.098830 6403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "12ca5c54d362408b872bea60d6ad784f"
format_stamp: "Formatted at 2024-05-01 07:59:58 on dist-test-slave-wslg"
I20240501 07:59:58.099319 6403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "12ca5c54d362408b872bea60d6ad784f"
format_stamp: "Formatted at 2024-05-01 07:59:58 on dist-test-slave-wslg"
I20240501 07:59:58.103137 6403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 07:59:58.105530 6418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:58.106680 6403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:58.106784 6403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "12ca5c54d362408b872bea60d6ad784f"
format_stamp: "Formatted at 2024-05-01 07:59:58 on dist-test-slave-wslg"
I20240501 07:59:58.106884 6403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:58.136705 6403 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:58.137524 6403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:58.137699 6403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:58.138362 6403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:58.139442 6403 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:58.139503 6403 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:58.139562 6403 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:58.139587 6403 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:58.148851 6403 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:40793
I20240501 07:59:58.148865 6531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:40793 every 8 connection(s)
I20240501 07:59:58.149896 6403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 07:59:58.153925 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 6403
I20240501 07:59:58.154088 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 07:59:58.159837 6532 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44911
I20240501 07:59:58.160127 6532 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:58.160893 6532 heartbeater.cc:511] Master 127.0.99.190:44911 requested a full tablet report, sending...
I20240501 07:59:58.162223 3534 ts_manager.cc:194] Registered new tserver with Master: 12ca5c54d362408b872bea60d6ad784f (127.0.99.136:40793)
I20240501 07:59:58.163048 3534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:44279
I20240501 07:59:58.365139 5268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:58.367120 3790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:58.367427 3658 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:58.373235 5006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.374442 6466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.374826 5137 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:58.388422 4871 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:58.390167 3920 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:58.497601 3658 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:58.499058 5268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:58.499526 6466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.508706 3790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:58.508922 3920 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:58.514218 4871 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:58.514266 5137 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:58.544770 5006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.575719 3806 consensus_queue.cc:237] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 231, Committed index: 231, Last appended: 1.231, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } }
I20240501 07:59:58.577229 5024 raft_consensus.cc:1261] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 231. Preceding OpId from leader: term: 1 index: 232. (index mismatch)
I20240501 07:59:58.577728 6626 consensus_queue.cc:1035] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
I20240501 07:59:58.579717 6368 raft_consensus.cc:2932] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Committing config change with OpId 1.232: config changed from index 69 to 232, NON_VOTER 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } } }
I20240501 07:59:58.581845 5024 raft_consensus.cc:2932] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Committing config change with OpId 1.232: config changed from index 69 to 232, NON_VOTER 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } } }
W20240501 07:59:58.582536 3746 consensus_peers.cc:485] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e -> Peer 12ca5c54d362408b872bea60d6ad784f (127.0.99.136:40793): Couldn't send request to peer 12ca5c54d362408b872bea60d6ad784f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5432d22bf6dc4866bbc3d91f21a45c21. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:58.582511 3534 catalog_manager.cc:5484] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e reported cstate change: config changed from index 69 to 232, NON_VOTER 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:58.590961 4891 consensus_queue.cc:237] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 228, Committed index: 228, Last appended: 2.228, Last appended by leader: 57, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } }
I20240501 07:59:58.592556 3676 raft_consensus.cc:1261] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Refusing update from remote peer d8461c1ff1ed48b09047faeea87e9598: Log matching property violated. Preceding OpId in replica: term: 2 index: 228. Preceding OpId from leader: term: 2 index: 229. (index mismatch)
I20240501 07:59:58.592947 6640 consensus_queue.cc:1035] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 228, Time since last communication: 0.000s
I20240501 07:59:58.595156 6640 raft_consensus.cc:2932] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [term 2 LEADER]: Committing config change with OpId 2.229: config changed from index 60 to 229, NON_VOTER 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) added. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } } }
I20240501 07:59:58.595499 3676 raft_consensus.cc:2932] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Committing config change with OpId 2.229: config changed from index 60 to 229, NON_VOTER 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) added. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } } }
W20240501 07:59:58.597563 4828 consensus_peers.cc:485] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 -> Peer 12ca5c54d362408b872bea60d6ad784f (127.0.99.136:40793): Couldn't send request to peer 12ca5c54d362408b872bea60d6ad784f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31a0bc2519cd43eda2f2fc8b5a6e7d5b. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:58.597776 3531 catalog_manager.cc:5484] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 reported cstate change: config changed from index 60 to 229, NON_VOTER 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "d8461c1ff1ed48b09047faeea87e9598" committed_config { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:58.606287 3940 consensus_queue.cc:237] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 1.282, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } }
I20240501 07:59:58.607775 5281 raft_consensus.cc:1261] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
I20240501 07:59:58.608206 6649 consensus_queue.cc:1035] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20240501 07:59:58.610018 6356 raft_consensus.cc:2932] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Committing config change with OpId 1.283: config changed from index 82 to 283, NON_VOTER 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) added. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } } }
I20240501 07:59:58.611074 5281 raft_consensus.cc:2932] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 82 to 283, NON_VOTER 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) added. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } } }
W20240501 07:59:58.611228 3877 consensus_peers.cc:485] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c -> Peer 12ca5c54d362408b872bea60d6ad784f (127.0.99.136:40793): Couldn't send request to peer 12ca5c54d362408b872bea60d6ad784f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc31e13bb07f44ddb7b80282ee4c434b. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:58.612649 3534 catalog_manager.cc:5484] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c reported cstate change: config changed from index 82 to 283, NON_VOTER 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:58.621085 5281 consensus_queue.cc:237] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } }
I20240501 07:59:58.622543 4891 raft_consensus.cc:1261] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Refusing update from remote peer 09d8fa5c011b4e398612ba0274c14f2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:58.624027 6280 consensus_queue.cc:1035] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:58.626013 6659 raft_consensus.cc:2932] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } } }
W20240501 07:59:58.626248 5224 consensus_peers.cc:485] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e -> Peer 12ca5c54d362408b872bea60d6ad784f (127.0.99.136:40793): Couldn't send request to peer 12ca5c54d362408b872bea60d6ad784f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71a2915d4efd4d48a1d41df5ecf2f4d7. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:58.626471 4891 raft_consensus.cc:2932] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } } }
I20240501 07:59:58.628880 3531 catalog_manager.cc:5484] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e reported cstate change: config changed from index -1 to 3, NON_VOTER 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "09d8fa5c011b4e398612ba0274c14f2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:59.084378 6707 ts_tablet_manager.cc:902] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f: Initiating tablet copy from peer d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132:37117)
I20240501 07:59:59.085094 6707 tablet_copy_client.cc:308] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37117
I20240501 07:59:59.088776 4913 tablet_copy_service.cc:140] P d8461c1ff1ed48b09047faeea87e9598: Received BeginTabletCopySession request for tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b from peer 12ca5c54d362408b872bea60d6ad784f ({username='slave'} at 127.0.99.136:48341)
I20240501 07:59:59.088903 4913 tablet_copy_service.cc:161] P d8461c1ff1ed48b09047faeea87e9598: Beginning new tablet copy session on tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b from peer 12ca5c54d362408b872bea60d6ad784f at {username='slave'} at 127.0.99.136:48341: session id = 12ca5c54d362408b872bea60d6ad784f-31a0bc2519cd43eda2f2fc8b5a6e7d5b
I20240501 07:59:59.090006 4913 tablet_copy_source_session.cc:205] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:59.093232 6707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:59.100116 6707 tablet_copy_client.cc:783] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:59.100370 6707 tablet_copy_client.cc:647] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:59.102283 6707 tablet_copy_client.cc:523] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:59.104106 6707 tablet_bootstrap.cc:492] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f: Bootstrap starting.
I20240501 07:59:59.122092 6707 log.cc:824] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:59.143322 6730 ts_tablet_manager.cc:902] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f: Initiating tablet copy from peer 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135:41559)
I20240501 07:59:59.143901 6730 tablet_copy_client.cc:308] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:41559
I20240501 07:59:59.147234 5303 tablet_copy_service.cc:140] P 09d8fa5c011b4e398612ba0274c14f2e: Received BeginTabletCopySession request for tablet 71a2915d4efd4d48a1d41df5ecf2f4d7 from peer 12ca5c54d362408b872bea60d6ad784f ({username='slave'} at 127.0.99.136:58739)
I20240501 07:59:59.147377 5303 tablet_copy_service.cc:161] P 09d8fa5c011b4e398612ba0274c14f2e: Beginning new tablet copy session on tablet 71a2915d4efd4d48a1d41df5ecf2f4d7 from peer 12ca5c54d362408b872bea60d6ad784f at {username='slave'} at 127.0.99.136:58739: session id = 12ca5c54d362408b872bea60d6ad784f-71a2915d4efd4d48a1d41df5ecf2f4d7
I20240501 07:59:59.148540 5303 tablet_copy_source_session.cc:205] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:59.149596 6730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a2915d4efd4d48a1d41df5ecf2f4d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:59.151316 6730 tablet_copy_client.cc:783] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:59.151465 6730 tablet_copy_client.cc:647] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:59.152631 6730 tablet_copy_client.cc:523] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:59.153534 6730 tablet_bootstrap.cc:492] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f: Bootstrap starting.
I20240501 07:59:59.157192 6730 tablet_bootstrap.cc:492] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:59.157447 6730 tablet_bootstrap.cc:492] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f: Bootstrap complete.
I20240501 07:59:59.157713 6730 ts_tablet_manager.cc:1367] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 07:59:59.160709 6730 raft_consensus.cc:348] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } }
I20240501 07:59:59.161020 6730 raft_consensus.cc:729] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 12ca5c54d362408b872bea60d6ad784f, State: Initialized, Role: LEARNER
I20240501 07:59:59.161372 6730 consensus_queue.cc:260] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } }
I20240501 07:59:59.162138 6532 heartbeater.cc:503] Master 127.0.99.190:44911 was elected leader, sending a full tablet report...
I20240501 07:59:59.162329 6730 ts_tablet_manager.cc:1398] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:59:59.163026 5303 tablet_copy_service.cc:342] P 09d8fa5c011b4e398612ba0274c14f2e: Request end of tablet copy session 12ca5c54d362408b872bea60d6ad784f-71a2915d4efd4d48a1d41df5ecf2f4d7 received from {username='slave'} at 127.0.99.136:58739
I20240501 07:59:59.163128 5303 tablet_copy_service.cc:433] P 09d8fa5c011b4e398612ba0274c14f2e: ending tablet copy session 12ca5c54d362408b872bea60d6ad784f-71a2915d4efd4d48a1d41df5ecf2f4d7 on tablet 71a2915d4efd4d48a1d41df5ecf2f4d7 with peer 12ca5c54d362408b872bea60d6ad784f
I20240501 07:59:59.165628 6730 ts_tablet_manager.cc:902] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f: Initiating tablet copy from peer 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191)
I20240501 07:59:59.166106 6730 tablet_copy_client.cc:308] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43191
I20240501 07:59:59.169633 3829 tablet_copy_service.cc:140] P 40c9ccd5f29c424eb845d670a6f7d21e: Received BeginTabletCopySession request for tablet 5432d22bf6dc4866bbc3d91f21a45c21 from peer 12ca5c54d362408b872bea60d6ad784f ({username='slave'} at 127.0.99.136:44401)
I20240501 07:59:59.169742 3829 tablet_copy_service.cc:161] P 40c9ccd5f29c424eb845d670a6f7d21e: Beginning new tablet copy session on tablet 5432d22bf6dc4866bbc3d91f21a45c21 from peer 12ca5c54d362408b872bea60d6ad784f at {username='slave'} at 127.0.99.136:44401: session id = 12ca5c54d362408b872bea60d6ad784f-5432d22bf6dc4866bbc3d91f21a45c21
I20240501 07:59:59.170569 3829 tablet_copy_source_session.cc:205] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:59.171344 6730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5432d22bf6dc4866bbc3d91f21a45c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:59.172969 6730 tablet_copy_client.cc:783] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:59.173139 6730 tablet_copy_client.cc:647] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:59.175081 6730 tablet_copy_client.cc:523] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:59.176013 6730 tablet_bootstrap.cc:492] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f: Bootstrap starting.
I20240501 07:59:59.189648 6741 ts_tablet_manager.cc:902] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f: Initiating tablet copy from peer 8635309522a045b1941fea81699d524c (127.0.99.131:39655)
I20240501 07:59:59.190306 6741 tablet_copy_client.cc:308] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39655
I20240501 07:59:59.194458 3961 tablet_copy_service.cc:140] P 8635309522a045b1941fea81699d524c: Received BeginTabletCopySession request for tablet cc31e13bb07f44ddb7b80282ee4c434b from peer 12ca5c54d362408b872bea60d6ad784f ({username='slave'} at 127.0.99.136:33365)
I20240501 07:59:59.194569 3961 tablet_copy_service.cc:161] P 8635309522a045b1941fea81699d524c: Beginning new tablet copy session on tablet cc31e13bb07f44ddb7b80282ee4c434b from peer 12ca5c54d362408b872bea60d6ad784f at {username='slave'} at 127.0.99.136:33365: session id = 12ca5c54d362408b872bea60d6ad784f-cc31e13bb07f44ddb7b80282ee4c434b
I20240501 07:59:59.195362 3961 tablet_copy_source_session.cc:205] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:59.196168 6741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc31e13bb07f44ddb7b80282ee4c434b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:59.198230 6741 tablet_copy_client.cc:783] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:59.198388 6741 tablet_copy_client.cc:647] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:59.200243 6741 tablet_copy_client.cc:523] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:59.201622 6741 tablet_bootstrap.cc:492] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f: Bootstrap starting.
I20240501 07:59:59.264966 6707 tablet_bootstrap.cc:492] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f: Bootstrap replayed 1/1 log segments. Stats: ops{read=229 overwritten=0 applied=229 ignored=0} inserts{seen=856 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:59.265324 6707 tablet_bootstrap.cc:492] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f: Bootstrap complete.
I20240501 07:59:59.265535 6707 ts_tablet_manager.cc:1367] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f: Time spent bootstrapping tablet: real 0.162s user 0.142s sys 0.017s
I20240501 07:59:59.266147 6707 raft_consensus.cc:348] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } }
I20240501 07:59:59.266264 6707 raft_consensus.cc:729] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 12ca5c54d362408b872bea60d6ad784f, State: Initialized, Role: LEARNER
I20240501 07:59:59.266378 6707 consensus_queue.cc:260] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 229, Last appended: 2.229, Last appended by leader: 229, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } }
I20240501 07:59:59.266645 6707 ts_tablet_manager.cc:1398] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 07:59:59.267418 4913 tablet_copy_service.cc:342] P d8461c1ff1ed48b09047faeea87e9598: Request end of tablet copy session 12ca5c54d362408b872bea60d6ad784f-31a0bc2519cd43eda2f2fc8b5a6e7d5b received from {username='slave'} at 127.0.99.136:48341
I20240501 07:59:59.267500 4913 tablet_copy_service.cc:433] P d8461c1ff1ed48b09047faeea87e9598: ending tablet copy session 12ca5c54d362408b872bea60d6ad784f-31a0bc2519cd43eda2f2fc8b5a6e7d5b on tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b with peer 12ca5c54d362408b872bea60d6ad784f
I20240501 07:59:59.340395 6730 tablet_bootstrap.cc:492] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f: Bootstrap replayed 1/1 log segments. Stats: ops{read=232 overwritten=0 applied=232 ignored=0} inserts{seen=914 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:59.340760 6730 tablet_bootstrap.cc:492] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f: Bootstrap complete.
I20240501 07:59:59.340986 6730 ts_tablet_manager.cc:1367] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f: Time spent bootstrapping tablet: real 0.165s user 0.151s sys 0.009s
I20240501 07:59:59.341507 6730 raft_consensus.cc:348] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } }
I20240501 07:59:59.341640 6730 raft_consensus.cc:729] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 12ca5c54d362408b872bea60d6ad784f, State: Initialized, Role: LEARNER
I20240501 07:59:59.341754 6730 consensus_queue.cc:260] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 232, Last appended: 1.232, Last appended by leader: 232, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } }
I20240501 07:59:59.342204 6730 ts_tablet_manager.cc:1398] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 07:59:59.342797 3829 tablet_copy_service.cc:342] P 40c9ccd5f29c424eb845d670a6f7d21e: Request end of tablet copy session 12ca5c54d362408b872bea60d6ad784f-5432d22bf6dc4866bbc3d91f21a45c21 received from {username='slave'} at 127.0.99.136:44401
I20240501 07:59:59.342888 3829 tablet_copy_service.cc:433] P 40c9ccd5f29c424eb845d670a6f7d21e: ending tablet copy session 12ca5c54d362408b872bea60d6ad784f-5432d22bf6dc4866bbc3d91f21a45c21 on tablet 5432d22bf6dc4866bbc3d91f21a45c21 with peer 12ca5c54d362408b872bea60d6ad784f
I20240501 07:59:59.405308 6741 tablet_bootstrap.cc:492] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f: Bootstrap replayed 1/1 log segments. Stats: ops{read=283 overwritten=0 applied=283 ignored=0} inserts{seen=911 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:59.405632 6741 tablet_bootstrap.cc:492] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f: Bootstrap complete.
I20240501 07:59:59.405835 6741 ts_tablet_manager.cc:1367] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f: Time spent bootstrapping tablet: real 0.204s user 0.183s sys 0.013s
I20240501 07:59:59.406280 6741 raft_consensus.cc:348] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } }
I20240501 07:59:59.406388 6741 raft_consensus.cc:729] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 12ca5c54d362408b872bea60d6ad784f, State: Initialized, Role: LEARNER
I20240501 07:59:59.406476 6741 consensus_queue.cc:260] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 283, Last appended: 1.283, Last appended by leader: 283, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: true } }
I20240501 07:59:59.406735 6741 ts_tablet_manager.cc:1398] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 07:59:59.407341 3961 tablet_copy_service.cc:342] P 8635309522a045b1941fea81699d524c: Request end of tablet copy session 12ca5c54d362408b872bea60d6ad784f-cc31e13bb07f44ddb7b80282ee4c434b received from {username='slave'} at 127.0.99.136:33365
I20240501 07:59:59.407436 3961 tablet_copy_service.cc:433] P 8635309522a045b1941fea81699d524c: ending tablet copy session 12ca5c54d362408b872bea60d6ad784f-cc31e13bb07f44ddb7b80282ee4c434b on tablet cc31e13bb07f44ddb7b80282ee4c434b with peer 12ca5c54d362408b872bea60d6ad784f
I20240501 07:59:59.517889 6486 raft_consensus.cc:1203] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.228->[2.229-2.229] Dedup: 2.229->[]
I20240501 07:59:59.601433 6486 raft_consensus.cc:1203] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.231->[1.232-1.232] Dedup: 1.232->[]
I20240501 07:59:59.736768 6486 raft_consensus.cc:1203] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240501 07:59:59.759856 6486 raft_consensus.cc:1203] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.282->[1.283-1.283] Dedup: 1.283->[]
I20240501 08:00:00.024040 6791 raft_consensus.cc:1050] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e: attempting to promote NON_VOTER 12ca5c54d362408b872bea60d6ad784f to VOTER
I20240501 08:00:00.024464 6791 consensus_queue.cc:237] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 232, Committed index: 232, Last appended: 1.232, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } }
I20240501 08:00:00.025827 5024 raft_consensus.cc:1261] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 232. Preceding OpId from leader: term: 1 index: 233. (index mismatch)
I20240501 08:00:00.025909 6486 raft_consensus.cc:1261] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [term 1 LEARNER]: Refusing update from remote peer 40c9ccd5f29c424eb845d670a6f7d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 232. Preceding OpId from leader: term: 1 index: 233. (index mismatch)
I20240501 08:00:00.026345 6791 consensus_queue.cc:1035] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 232, Time since last communication: 0.000s
I20240501 08:00:00.026554 6368 consensus_queue.cc:1035] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 232, Time since last communication: 0.000s
I20240501 08:00:00.028474 6368 raft_consensus.cc:2932] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Committing config change with OpId 1.233: config changed from index 232 to 233, 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.030117 6486 raft_consensus.cc:2932] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [term 1 FOLLOWER]: Committing config change with OpId 1.233: config changed from index 232 to 233, 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.030663 5024 raft_consensus.cc:2932] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Committing config change with OpId 1.233: config changed from index 232 to 233, 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.031133 3534 catalog_manager.cc:5484] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e reported cstate change: config changed from index 232 to 233, 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" committed_config { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:00.061712 6768 raft_consensus.cc:1050] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598: attempting to promote NON_VOTER 12ca5c54d362408b872bea60d6ad784f to VOTER
I20240501 08:00:00.062224 6768 consensus_queue.cc:237] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 229, Committed index: 229, Last appended: 2.229, Last appended by leader: 57, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } }
I20240501 08:00:00.063566 6486 raft_consensus.cc:1261] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f [term 2 LEARNER]: Refusing update from remote peer d8461c1ff1ed48b09047faeea87e9598: Log matching property violated. Preceding OpId in replica: term: 2 index: 229. Preceding OpId from leader: term: 2 index: 230. (index mismatch)
I20240501 08:00:00.063611 3676 raft_consensus.cc:1261] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Refusing update from remote peer d8461c1ff1ed48b09047faeea87e9598: Log matching property violated. Preceding OpId in replica: term: 2 index: 229. Preceding OpId from leader: term: 2 index: 230. (index mismatch)
I20240501 08:00:00.063957 6768 consensus_queue.cc:1035] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 230, Last known committed idx: 229, Time since last communication: 0.000s
I20240501 08:00:00.064158 6846 consensus_queue.cc:1035] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 230, Last known committed idx: 229, Time since last communication: 0.000s
I20240501 08:00:00.066102 6848 raft_consensus.cc:2932] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [term 2 LEADER]: Committing config change with OpId 2.230: config changed from index 229 to 230, 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.066502 3676 raft_consensus.cc:2932] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Committing config change with OpId 2.230: config changed from index 229 to 230, 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.066529 6486 raft_consensus.cc:2932] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f [term 2 FOLLOWER]: Committing config change with OpId 2.230: config changed from index 229 to 230, 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.069103 3536 catalog_manager.cc:5484] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad reported cstate change: config changed from index 229 to 230, 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d8461c1ff1ed48b09047faeea87e9598" committed_config { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "014bc50d0c5a44bdbf49c0eb41fa93ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42453 } attrs { replace: true } } peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.072438 4891 consensus_queue.cc:237] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 230, Committed index: 230, Last appended: 2.230, Last appended by leader: 57, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } }
I20240501 08:00:00.073415 6486 raft_consensus.cc:1261] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f [term 2 FOLLOWER]: Refusing update from remote peer d8461c1ff1ed48b09047faeea87e9598: Log matching property violated. Preceding OpId in replica: term: 2 index: 230. Preceding OpId from leader: term: 2 index: 231. (index mismatch)
I20240501 08:00:00.073781 6851 consensus_queue.cc:1035] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 231, Last known committed idx: 230, Time since last communication: 0.000s
I20240501 08:00:00.075174 6768 raft_consensus.cc:2932] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P d8461c1ff1ed48b09047faeea87e9598 [term 2 LEADER]: Committing config change with OpId 2.231: config changed from index 230 to 231, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.075618 6486 raft_consensus.cc:2932] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f [term 2 FOLLOWER]: Committing config change with OpId 2.231: config changed from index 230 to 231, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.076663 3523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b with cas_config_opid_index 230: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:00.077951 3531 catalog_manager.cc:5484] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 12ca5c54d362408b872bea60d6ad784f reported cstate change: config changed from index 230 to 231, VOTER 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "d8461c1ff1ed48b09047faeea87e9598" committed_config { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.080132 3658 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b with delete_type TABLET_DATA_TOMBSTONED (TS 014bc50d0c5a44bdbf49c0eb41fa93ad not found in new config with opid_index 231) from {username='slave'} at 127.0.0.1:46686
I20240501 08:00:00.080513 6852 tablet_replica.cc:330] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad: stopping tablet replica
I20240501 08:00:00.080644 6852 raft_consensus.cc:2229] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:00.080801 6852 raft_consensus.cc:2258] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:00.082175 6852 ts_tablet_manager.cc:1875] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:00.084030 6852 ts_tablet_manager.cc:1888] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.230
I20240501 08:00:00.084102 6852 log.cc:1196] T 31a0bc2519cd43eda2f2fc8b5a6e7d5b P 014bc50d0c5a44bdbf49c0eb41fa93ad: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/31a0bc2519cd43eda2f2fc8b5a6e7d5b
I20240501 08:00:00.084654 3523 catalog_manager.cc:4830] TS 014bc50d0c5a44bdbf49c0eb41fa93ad (127.0.99.129:42453): tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b (table rebalance_test_table_4 [id=980322bb0877411f904d1f982f67275b]) successfully deleted
I20240501 08:00:00.165046 6652 raft_consensus.cc:1050] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c: attempting to promote NON_VOTER 12ca5c54d362408b872bea60d6ad784f to VOTER
I20240501 08:00:00.165498 6652 consensus_queue.cc:237] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 1.283, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } }
I20240501 08:00:00.166855 6486 raft_consensus.cc:1261] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [term 1 LEARNER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 283. Preceding OpId from leader: term: 1 index: 284. (index mismatch)
I20240501 08:00:00.166913 5281 raft_consensus.cc:1261] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Refusing update from remote peer 8635309522a045b1941fea81699d524c: Log matching property violated. Preceding OpId in replica: term: 1 index: 283. Preceding OpId from leader: term: 1 index: 284. (index mismatch)
I20240501 08:00:00.167318 6800 consensus_queue.cc:1035] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I20240501 08:00:00.167531 6652 consensus_queue.cc:1035] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I20240501 08:00:00.169390 6854 raft_consensus.cc:2932] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Committing config change with OpId 1.284: config changed from index 283 to 284, 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.170043 6486 raft_consensus.cc:2932] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [term 1 FOLLOWER]: Committing config change with OpId 1.284: config changed from index 283 to 284, 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.170200 5281 raft_consensus.cc:2932] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Committing config change with OpId 1.284: config changed from index 283 to 284, 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.172570 3534 catalog_manager.cc:5484] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f reported cstate change: config changed from index 283 to 284, 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8635309522a045b1941fea81699d524c" committed_config { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.216929 3940 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc31e13bb07f44ddb7b80282ee4c434b"
dest_uuid: "8635309522a045b1941fea81699d524c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42412
I20240501 08:00:00.217075 3940 raft_consensus.cc:595] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:00.227628 3806 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5432d22bf6dc4866bbc3d91f21a45c21"
dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35880
I20240501 08:00:00.227819 3806 raft_consensus.cc:595] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:00.249711 3940 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc31e13bb07f44ddb7b80282ee4c434b"
dest_uuid: "8635309522a045b1941fea81699d524c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42432
I20240501 08:00:00.249881 3940 raft_consensus.cc:595] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:00.261111 3806 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5432d22bf6dc4866bbc3d91f21a45c21"
dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35908
I20240501 08:00:00.261258 3806 raft_consensus.cc:595] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:00.327255 6912 raft_consensus.cc:1050] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e: attempting to promote NON_VOTER 12ca5c54d362408b872bea60d6ad784f to VOTER
I20240501 08:00:00.327740 6912 consensus_queue.cc:237] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } }
I20240501 08:00:00.329177 6486 raft_consensus.cc:1261] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f [term 2 LEARNER]: Refusing update from remote peer 09d8fa5c011b4e398612ba0274c14f2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:00:00.329178 4891 raft_consensus.cc:1261] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Refusing update from remote peer 09d8fa5c011b4e398612ba0274c14f2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:00:00.329572 6912 consensus_queue.cc:1035] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:00.329749 6914 consensus_queue.cc:1035] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:00.331627 6912 raft_consensus.cc:2932] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.332077 6486 raft_consensus.cc:2932] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.335137 3536 catalog_manager.cc:5484] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e reported cstate change: config changed from index 3 to 4, 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "09d8fa5c011b4e398612ba0274c14f2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:00.341645 5281 consensus_queue.cc:237] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } }
I20240501 08:00:00.342243 4891 raft_consensus.cc:2932] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 12ca5c54d362408b872bea60d6ad784f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8461c1ff1ed48b09047faeea87e9598" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37117 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.342769 6486 raft_consensus.cc:1261] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f [term 2 FOLLOWER]: Refusing update from remote peer 09d8fa5c011b4e398612ba0274c14f2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:00.343147 6792 consensus_queue.cc:1035] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:00.345635 6912 raft_consensus.cc:2932] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.347451 3522 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71a2915d4efd4d48a1d41df5ecf2f4d7 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:00.347548 6486 raft_consensus.cc:2932] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 12ca5c54d362408b872bea60d6ad784f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:00.348141 3536 catalog_manager.cc:5484] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P 09d8fa5c011b4e398612ba0274c14f2e reported cstate change: config changed from index 4 to 5, VOTER d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "09d8fa5c011b4e398612ba0274c14f2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:00.350781 4871 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a2915d4efd4d48a1d41df5ecf2f4d7 with delete_type TABLET_DATA_TOMBSTONED (TS d8461c1ff1ed48b09047faeea87e9598 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44172
I20240501 08:00:00.351305 6918 tablet_replica.cc:330] stopping tablet replica
I20240501 08:00:00.351788 6918 raft_consensus.cc:2229] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:00.352025 6918 raft_consensus.cc:2258] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:00.353952 6918 ts_tablet_manager.cc:1875] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:00.356024 6918 ts_tablet_manager.cc:1888] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 08:00:00.356110 6918 log.cc:1196] T 71a2915d4efd4d48a1d41df5ecf2f4d7 P d8461c1ff1ed48b09047faeea87e9598: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/71a2915d4efd4d48a1d41df5ecf2f4d7
I20240501 08:00:00.356657 3523 catalog_manager.cc:4830] TS d8461c1ff1ed48b09047faeea87e9598 (127.0.99.132:37117): tablet 71a2915d4efd4d48a1d41df5ecf2f4d7 (table TestTable [id=3af7e8ba491243f4b3f46209249cc0ab]) successfully deleted
I20240501 08:00:00.431597 6368 raft_consensus.cc:979] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e: : Instructing follower 12ca5c54d362408b872bea60d6ad784f to start an election
I20240501 08:00:00.431735 6368 raft_consensus.cc:1067] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Signalling peer 12ca5c54d362408b872bea60d6ad784f to start an election
I20240501 08:00:00.432250 6486 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5432d22bf6dc4866bbc3d91f21a45c21"
dest_uuid: "12ca5c54d362408b872bea60d6ad784f"
from {username='slave'} at 127.0.99.130:35063
I20240501 08:00:00.432466 6486 raft_consensus.cc:482] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:00.432580 6486 raft_consensus.cc:3037] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:00.433455 6486 raft_consensus.cc:504] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } }
I20240501 08:00:00.434535 6486 leader_election.cc:288] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [CANDIDATE]: Term 2 election: Requested vote from peers 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191), da103aa972ab49889fc0bce4abc56bf7 (127.0.99.133:40225)
I20240501 08:00:00.434917 3806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5432d22bf6dc4866bbc3d91f21a45c21" candidate_uuid: "12ca5c54d362408b872bea60d6ad784f" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: true dest_uuid: "40c9ccd5f29c424eb845d670a6f7d21e"
I20240501 08:00:00.435055 3806 raft_consensus.cc:3032] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:00.435114 3806 raft_consensus.cc:729] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 40c9ccd5f29c424eb845d670a6f7d21e, State: Running, Role: LEADER
I20240501 08:00:00.435281 3806 consensus_queue.cc:260] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 1.233, Last appended by leader: 233, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } }
I20240501 08:00:00.435483 3806 raft_consensus.cc:3037] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:00.436525 3806 raft_consensus.cc:2454] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12ca5c54d362408b872bea60d6ad784f in term 2.
I20240501 08:00:00.436923 6421 leader_election.cc:302] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12ca5c54d362408b872bea60d6ad784f, 40c9ccd5f29c424eb845d670a6f7d21e; no voters:
I20240501 08:00:00.437171 6736 raft_consensus.cc:2781] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:00.437402 6736 raft_consensus.cc:686] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [term 2 LEADER]: Becoming Leader. State: Replica: 12ca5c54d362408b872bea60d6ad784f, State: Running, Role: LEADER
I20240501 08:00:00.437706 6736 consensus_queue.cc:237] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 1.233, Last appended by leader: 233, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } }
I20240501 08:00:00.438831 5024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5432d22bf6dc4866bbc3d91f21a45c21" candidate_uuid: "12ca5c54d362408b872bea60d6ad784f" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: true dest_uuid: "da103aa972ab49889fc0bce4abc56bf7"
I20240501 08:00:00.439018 5024 raft_consensus.cc:3037] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:00.440158 5024 raft_consensus.cc:2454] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12ca5c54d362408b872bea60d6ad784f in term 2.
I20240501 08:00:00.440613 3531 catalog_manager.cc:5484] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f reported cstate change: term changed from 1 to 2, leader changed from 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) to 12ca5c54d362408b872bea60d6ad784f (127.0.99.136). New cstate: current_term: 2 leader_uuid: "12ca5c54d362408b872bea60d6ad784f" committed_config { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:00.478062 3940 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc31e13bb07f44ddb7b80282ee4c434b"
dest_uuid: "8635309522a045b1941fea81699d524c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42452
I20240501 08:00:00.478237 3940 raft_consensus.cc:595] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:00.496908 3940 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc31e13bb07f44ddb7b80282ee4c434b"
dest_uuid: "8635309522a045b1941fea81699d524c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42476
I20240501 08:00:00.497134 3940 raft_consensus.cc:595] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:00.622996 6652 raft_consensus.cc:979] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c: : Instructing follower 12ca5c54d362408b872bea60d6ad784f to start an election
I20240501 08:00:00.623143 6652 raft_consensus.cc:1067] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Signalling peer 12ca5c54d362408b872bea60d6ad784f to start an election
I20240501 08:00:00.623672 6486 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cc31e13bb07f44ddb7b80282ee4c434b"
dest_uuid: "12ca5c54d362408b872bea60d6ad784f"
from {username='slave'} at 127.0.99.131:58115
I20240501 08:00:00.623837 6486 raft_consensus.cc:482] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:00.623911 6486 raft_consensus.cc:3037] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:00.624817 6486 raft_consensus.cc:504] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } }
I20240501 08:00:00.625389 6486 leader_election.cc:288] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [CANDIDATE]: Term 2 election: Requested vote from peers 8635309522a045b1941fea81699d524c (127.0.99.131:39655), 09d8fa5c011b4e398612ba0274c14f2e (127.0.99.135:41559)
I20240501 08:00:00.625780 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc31e13bb07f44ddb7b80282ee4c434b" candidate_uuid: "12ca5c54d362408b872bea60d6ad784f" candidate_term: 2 candidate_status { last_received { term: 1 index: 284 } } ignore_live_leader: true dest_uuid: "8635309522a045b1941fea81699d524c"
I20240501 08:00:00.625838 5279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc31e13bb07f44ddb7b80282ee4c434b" candidate_uuid: "12ca5c54d362408b872bea60d6ad784f" candidate_term: 2 candidate_status { last_received { term: 1 index: 284 } } ignore_live_leader: true dest_uuid: "09d8fa5c011b4e398612ba0274c14f2e"
I20240501 08:00:00.625917 3940 raft_consensus.cc:3032] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:00.625958 5279 raft_consensus.cc:3037] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:00.625972 3940 raft_consensus.cc:729] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8635309522a045b1941fea81699d524c, State: Running, Role: LEADER
I20240501 08:00:00.626173 3940 consensus_queue.cc:260] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 1.284, Last appended by leader: 284, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } }
I20240501 08:00:00.626438 3940 raft_consensus.cc:3037] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:00.626922 5279 raft_consensus.cc:2454] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12ca5c54d362408b872bea60d6ad784f in term 2.
I20240501 08:00:00.627242 6420 leader_election.cc:302] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09d8fa5c011b4e398612ba0274c14f2e, 12ca5c54d362408b872bea60d6ad784f; no voters:
I20240501 08:00:00.627283 3940 raft_consensus.cc:2454] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12ca5c54d362408b872bea60d6ad784f in term 2.
I20240501 08:00:00.627426 6736 raft_consensus.cc:2781] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:00.627524 6736 raft_consensus.cc:686] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [term 2 LEADER]: Becoming Leader. State: Replica: 12ca5c54d362408b872bea60d6ad784f, State: Running, Role: LEADER
I20240501 08:00:00.627681 6736 consensus_queue.cc:237] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 1.284, Last appended by leader: 284, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } }
I20240501 08:00:00.629462 3531 catalog_manager.cc:5484] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f reported cstate change: term changed from 1 to 2, leader changed from 8635309522a045b1941fea81699d524c (127.0.99.131) to 12ca5c54d362408b872bea60d6ad784f (127.0.99.136). New cstate: current_term: 2 leader_uuid: "12ca5c54d362408b872bea60d6ad784f" committed_config { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:00.820711 3806 raft_consensus.cc:1261] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 2 FOLLOWER]: Refusing update from remote peer 12ca5c54d362408b872bea60d6ad784f: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 2 index: 234. (index mismatch)
I20240501 08:00:00.821274 6736 consensus_queue.cc:1035] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c9ccd5f29c424eb845d670a6f7d21e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20240501 08:00:00.824424 5024 raft_consensus.cc:1261] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [term 2 FOLLOWER]: Refusing update from remote peer 12ca5c54d362408b872bea60d6ad784f: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 2 index: 234. (index mismatch)
I20240501 08:00:00.824806 6969 consensus_queue.cc:1035] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20240501 08:00:01.123041 3940 raft_consensus.cc:1261] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Refusing update from remote peer 12ca5c54d362408b872bea60d6ad784f: Log matching property violated. Preceding OpId in replica: term: 1 index: 284. Preceding OpId from leader: term: 2 index: 285. (index mismatch)
I20240501 08:00:01.123596 6969 consensus_queue.cc:1035] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8635309522a045b1941fea81699d524c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20240501 08:00:01.125895 5279 raft_consensus.cc:1261] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [term 2 FOLLOWER]: Refusing update from remote peer 12ca5c54d362408b872bea60d6ad784f: Log matching property violated. Preceding OpId in replica: term: 1 index: 284. Preceding OpId from leader: term: 2 index: 285. (index mismatch)
I20240501 08:00:01.126469 6736 consensus_queue.cc:1035] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20240501 08:00:01.285594 6486 consensus_queue.cc:237] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 2.234, Last appended by leader: 233, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } }
I20240501 08:00:01.286764 5024 raft_consensus.cc:1261] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [term 2 FOLLOWER]: Refusing update from remote peer 12ca5c54d362408b872bea60d6ad784f: Log matching property violated. Preceding OpId in replica: term: 2 index: 234. Preceding OpId from leader: term: 2 index: 235. (index mismatch)
I20240501 08:00:01.287173 6736 consensus_queue.cc:1035] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240501 08:00:01.289223 6921 raft_consensus.cc:2932] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f [term 2 LEADER]: Committing config change with OpId 2.235: config changed from index 233 to 235, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:01.289645 5024 raft_consensus.cc:2932] T 5432d22bf6dc4866bbc3d91f21a45c21 P da103aa972ab49889fc0bce4abc56bf7 [term 2 FOLLOWER]: Committing config change with OpId 2.235: config changed from index 233 to 235, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:01.290812 3524 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5432d22bf6dc4866bbc3d91f21a45c21 with cas_config_opid_index 233: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:01.291848 3531 catalog_manager.cc:5484] T 5432d22bf6dc4866bbc3d91f21a45c21 P 12ca5c54d362408b872bea60d6ad784f reported cstate change: config changed from index 233 to 235, VOTER 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "12ca5c54d362408b872bea60d6ad784f" committed_config { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "da103aa972ab49889fc0bce4abc56bf7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40225 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:01.294766 3790 tablet_service.cc:1514] Processing DeleteTablet for tablet 5432d22bf6dc4866bbc3d91f21a45c21 with delete_type TABLET_DATA_TOMBSTONED (TS 40c9ccd5f29c424eb845d670a6f7d21e not found in new config with opid_index 235) from {username='slave'} at 127.0.0.1:43228
I20240501 08:00:01.295126 6999 tablet_replica.cc:330] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e: stopping tablet replica
I20240501 08:00:01.295259 6999 raft_consensus.cc:2229] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:01.295405 6999 raft_consensus.cc:2258] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:01.296931 6999 ts_tablet_manager.cc:1875] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:01.299114 6999 ts_tablet_manager.cc:1888] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.234
I20240501 08:00:01.299191 6999 log.cc:1196] T 5432d22bf6dc4866bbc3d91f21a45c21 P 40c9ccd5f29c424eb845d670a6f7d21e: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5432d22bf6dc4866bbc3d91f21a45c21
I20240501 08:00:01.299811 3523 catalog_manager.cc:4830] TS 40c9ccd5f29c424eb845d670a6f7d21e (127.0.99.130:43191): tablet 5432d22bf6dc4866bbc3d91f21a45c21 (table rebalance_test_table_1 [id=4c0194cc5d4f4b1587cce57551ae4de8]) successfully deleted
I20240501 08:00:01.692504 6486 consensus_queue.cc:237] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 285, Committed index: 285, Last appended: 2.285, Last appended by leader: 284, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } }
I20240501 08:00:01.693631 5279 raft_consensus.cc:1261] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [term 2 FOLLOWER]: Refusing update from remote peer 12ca5c54d362408b872bea60d6ad784f: Log matching property violated. Preceding OpId in replica: term: 2 index: 285. Preceding OpId from leader: term: 2 index: 286. (index mismatch)
I20240501 08:00:01.694131 6921 consensus_queue.cc:1035] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 285, Time since last communication: 0.000s
I20240501 08:00:01.695920 6969 raft_consensus.cc:2932] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f [term 2 LEADER]: Committing config change with OpId 2.286: config changed from index 284 to 286, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:01.696396 5279 raft_consensus.cc:2932] T cc31e13bb07f44ddb7b80282ee4c434b P 09d8fa5c011b4e398612ba0274c14f2e [term 2 FOLLOWER]: Committing config change with OpId 2.286: config changed from index 284 to 286, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } } }
I20240501 08:00:01.697566 3524 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cc31e13bb07f44ddb7b80282ee4c434b with cas_config_opid_index 284: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:01.698943 3536 catalog_manager.cc:5484] T cc31e13bb07f44ddb7b80282ee4c434b P 12ca5c54d362408b872bea60d6ad784f reported cstate change: config changed from index 284 to 286, VOTER 8635309522a045b1941fea81699d524c (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "12ca5c54d362408b872bea60d6ad784f" committed_config { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "09d8fa5c011b4e398612ba0274c14f2e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12ca5c54d362408b872bea60d6ad784f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:01.701481 3920 tablet_service.cc:1514] Processing DeleteTablet for tablet cc31e13bb07f44ddb7b80282ee4c434b with delete_type TABLET_DATA_TOMBSTONED (TS 8635309522a045b1941fea81699d524c not found in new config with opid_index 286) from {username='slave'} at 127.0.0.1:53920
I20240501 08:00:01.701854 7021 tablet_replica.cc:330] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c: stopping tablet replica
I20240501 08:00:01.702056 7021 raft_consensus.cc:2229] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:01.702179 7021 raft_consensus.cc:2258] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:01.703689 7021 ts_tablet_manager.cc:1875] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:01.705783 7021 ts_tablet_manager.cc:1888] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.285
I20240501 08:00:01.705859 7021 log.cc:1196] T cc31e13bb07f44ddb7b80282ee4c434b P 8635309522a045b1941fea81699d524c: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/cc31e13bb07f44ddb7b80282ee4c434b
I20240501 08:00:01.706481 3523 catalog_manager.cc:4830] TS 8635309522a045b1941fea81699d524c (127.0.99.131:39655): tablet cc31e13bb07f44ddb7b80282ee4c434b (table rebalance_test_table_2 [id=712e7be85ad54a6ebf02a160dca63546]) successfully deleted
I20240501 08:00:01.846346 5268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:01.848408 6466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:01.850121 3658 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:00:01.864177 3790 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:00:01.870935 3920 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:01.876717 5137 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:01.884333 5006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.891299 4871 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:01.980561 6466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:01.981184 5268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:01.987002 3658 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:00:01.987329 5137 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:01.990998 5006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.994171 3790 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:00:02.014087 4871 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:02.014819 3920 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:02.111510 6466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:02.114041 5268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:02.117816 3920 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:02.122627 3790 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:00:02.124902 3658 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:00:02.127889 5137 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:02.134136 4871 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:02.147011 5006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
2033c6e9bc084ce3baa7be66008fcca6 | 127.0.99.190:44911 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:39583 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
014bc50d0c5a44bdbf49c0eb41fa93ad | 127.0.99.129:42453 | HEALTHY | <none> | 3 | 0
09d8fa5c011b4e398612ba0274c14f2e | 127.0.99.135:41559 | HEALTHY | <none> | 1 | 0
12ca5c54d362408b872bea60d6ad784f | 127.0.99.136:40793 | HEALTHY | <none> | 2 | 0
40c9ccd5f29c424eb845d670a6f7d21e | 127.0.99.130:43191 | HEALTHY | <none> | 3 | 0
6208df88dcc34b4daf3d27a574e53a95 | 127.0.99.134:36011 | HEALTHY | <none> | 1 | 0
8635309522a045b1941fea81699d524c | 127.0.99.131:39655 | HEALTHY | <none> | 4 | 0
d8461c1ff1ed48b09047faeea87e9598 | 127.0.99.132:37117 | HEALTHY | <none> | 2 | 0
da103aa972ab49889fc0bce4abc56bf7 | 127.0.99.133:40225 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:42453
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:43191
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:39655
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:37117
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:40225
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:36011
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:41559
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:40793
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:42453
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:43191
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:39655
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:37117
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:40225
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:36011
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:41559
server_dump_info_path | /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:40793
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:39583 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 08:00:02.183485 398 log_verifier.cc:126] Checking tablet 02f63ed2bd6345b686c74eb002bb306f
I20240501 08:00:02.228152 398 log_verifier.cc:177] Verified matching terms for 238 ops in tablet 02f63ed2bd6345b686c74eb002bb306f
I20240501 08:00:02.228268 398 log_verifier.cc:126] Checking tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b
I20240501 08:00:02.272887 398 log_verifier.cc:177] Verified matching terms for 231 ops in tablet 31a0bc2519cd43eda2f2fc8b5a6e7d5b
I20240501 08:00:02.273006 398 log_verifier.cc:126] Checking tablet 393b4eef45ff435abd0fde4aec22da88
I20240501 08:00:02.323539 398 log_verifier.cc:177] Verified matching terms for 265 ops in tablet 393b4eef45ff435abd0fde4aec22da88
I20240501 08:00:02.323652 398 log_verifier.cc:126] Checking tablet 5432d22bf6dc4866bbc3d91f21a45c21
I20240501 08:00:02.373387 398 log_verifier.cc:177] Verified matching terms for 235 ops in tablet 5432d22bf6dc4866bbc3d91f21a45c21
I20240501 08:00:02.373507 398 log_verifier.cc:126] Checking tablet 54a2b2f834494f29b306e80e48804a43
I20240501 08:00:02.416640 398 log_verifier.cc:177] Verified matching terms for 223 ops in tablet 54a2b2f834494f29b306e80e48804a43
I20240501 08:00:02.416762 398 log_verifier.cc:126] Checking tablet 71a2915d4efd4d48a1d41df5ecf2f4d7
I20240501 08:00:02.428815 398 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 71a2915d4efd4d48a1d41df5ecf2f4d7
I20240501 08:00:02.428887 398 log_verifier.cc:126] Checking tablet 72c5108aa1aa4cbd913d9ceb1a262d58
I20240501 08:00:02.469805 398 log_verifier.cc:177] Verified matching terms for 225 ops in tablet 72c5108aa1aa4cbd913d9ceb1a262d58
I20240501 08:00:02.469919 398 log_verifier.cc:126] Checking tablet 7c0cda2d32534bb29a6a2538ac13368a
I20240501 08:00:02.520437 398 log_verifier.cc:177] Verified matching terms for 263 ops in tablet 7c0cda2d32534bb29a6a2538ac13368a
I20240501 08:00:02.520550 398 log_verifier.cc:126] Checking tablet 839df5100c814d6bb78dc67a8c834a49
I20240501 08:00:02.556769 398 log_verifier.cc:177] Verified matching terms for 183 ops in tablet 839df5100c814d6bb78dc67a8c834a49
I20240501 08:00:02.556878 398 log_verifier.cc:126] Checking tablet a77d5e3c352f495cbee97be6a2c4661a
I20240501 08:00:02.595938 398 log_verifier.cc:177] Verified matching terms for 198 ops in tablet a77d5e3c352f495cbee97be6a2c4661a
I20240501 08:00:02.596045 398 log_verifier.cc:126] Checking tablet b23c9dc72fbc4e409f1d22506b11cb42
I20240501 08:00:02.634967 398 log_verifier.cc:177] Verified matching terms for 199 ops in tablet b23c9dc72fbc4e409f1d22506b11cb42
I20240501 08:00:02.635072 398 log_verifier.cc:126] Checking tablet c02a65bc31814809ae316ae5fdd07a6e
I20240501 08:00:02.675411 398 log_verifier.cc:177] Verified matching terms for 218 ops in tablet c02a65bc31814809ae316ae5fdd07a6e
I20240501 08:00:02.675523 398 log_verifier.cc:126] Checking tablet cc31e13bb07f44ddb7b80282ee4c434b
I20240501 08:00:02.729276 398 log_verifier.cc:177] Verified matching terms for 286 ops in tablet cc31e13bb07f44ddb7b80282ee4c434b
I20240501 08:00:02.729410 398 log_verifier.cc:126] Checking tablet ed30e32a4ba2483484b5bc5e3cfb44fd
I20240501 08:00:02.775424 398 log_verifier.cc:177] Verified matching terms for 253 ops in tablet ed30e32a4ba2483484b5bc5e3cfb44fd
I20240501 08:00:02.775550 398 log_verifier.cc:126] Checking tablet f1aa85993dc449b7b33ebccb4e907046
I20240501 08:00:02.819713 398 log_verifier.cc:177] Verified matching terms for 229 ops in tablet f1aa85993dc449b7b33ebccb4e907046
I20240501 08:00:02.819823 398 log_verifier.cc:126] Checking tablet f75acaf04c12462aa4b3fa7e019776d4
I20240501 08:00:02.864403 398 log_verifier.cc:177] Verified matching terms for 234 ops in tablet f75acaf04c12462aa4b3fa7e019776d4
I20240501 08:00:02.866135 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 3597
I20240501 08:00:02.874552 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 3728
I20240501 08:00:02.881601 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 3859
I20240501 08:00:02.888376 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 4810
I20240501 08:00:02.896651 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 4943
I20240501 08:00:02.902990 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 5075
I20240501 08:00:02.908902 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 5206
I20240501 08:00:02.915076 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 6403
I20240501 08:00:02.921061 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 3506
2024-05-01T08:00:02Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/3 (22487 ms)
[----------] 1 test from RebalanceParamTest (22487 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240501 08:00:02.976624 398 test_util.cc:268] Using random seed: -541586584
I20240501 08:00:02.977396 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:00:02.977437 398 ts_itest-base.cc:113] --------------
I20240501 08:00:02.977447 398 ts_itest-base.cc:114] 8 tablet servers
I20240501 08:00:02.977458 398 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:00:02.977468 398 ts_itest-base.cc:116] --------------
2024-05-01T08:00:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:00:02Z Disabled control of system clock
I20240501 08:00:02.985932 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:45395
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:45395
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 08:00:03.098412 7144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:03.098678 7144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:00:03.098721 7144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:03.101953 7144 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:00:03.102083 7144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:00:03.102110 7144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:00:03.102128 7144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:00:03.102146 7144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:00:03.106040 7144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:45395
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:45395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.7144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 08:00:03.107015 7144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:00:03.108183 7144 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:03.116470 7149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:03.117563 7152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:03.118489 7150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:03.119689 7144 server_base.cc:1034] running on GCE node
I20240501 08:00:03.120250 7144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:00:03.121256 7144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:00:03.122412 7144 hybrid_clock.cc:648] HybridClock initialized: now 1714550403122404 us; error 30 us; skew 500 ppm
I20240501 08:00:03.124548 7144 webserver.cc:460] Webserver started at http://127.0.99.190:39873/ using document root <none> and password file <none>
I20240501 08:00:03.125152 7144 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:03.125228 7144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:03.125530 7144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:00:03.127353 7144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6aa238de40d94e5f9898ac424a4d1f68"
format_stamp: "Formatted at 2024-05-01 08:00:03 on dist-test-slave-wslg"
I20240501 08:00:03.127851 7144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6aa238de40d94e5f9898ac424a4d1f68"
format_stamp: "Formatted at 2024-05-01 08:00:03 on dist-test-slave-wslg"
I20240501 08:00:03.131502 7144 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240501 08:00:03.133828 7158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:03.135027 7144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:00:03.135164 7144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6aa238de40d94e5f9898ac424a4d1f68"
format_stamp: "Formatted at 2024-05-01 08:00:03 on dist-test-slave-wslg"
I20240501 08:00:03.135273 7144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:03.151799 7144 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:03.152518 7144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:00:03.152683 7144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:03.159169 7144 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:45395
I20240501 08:00:03.159206 7210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:45395 every 8 connection(s)
I20240501 08:00:03.160171 7144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:00:03.161648 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 7144
I20240501 08:00:03.161801 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:00:03.163098 7211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:03.168558 7211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68: Bootstrap starting.
I20240501 08:00:03.170862 7211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:03.171584 7211 log.cc:824] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68: Log is configured to *not* fsync() on all Append() calls
I20240501 08:00:03.173193 7211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68: No bootstrap required, opened a new log
I20240501 08:00:03.175742 7211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa238de40d94e5f9898ac424a4d1f68" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45395 } }
I20240501 08:00:03.175920 7211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:03.175963 7211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aa238de40d94e5f9898ac424a4d1f68, State: Initialized, Role: FOLLOWER
I20240501 08:00:03.176396 7211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa238de40d94e5f9898ac424a4d1f68" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45395 } }
I20240501 08:00:03.176517 7211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:00:03.176578 7211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:00:03.176685 7211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:03.177465 7211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa238de40d94e5f9898ac424a4d1f68" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45395 } }
I20240501 08:00:03.177758 7211 leader_election.cc:302] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aa238de40d94e5f9898ac424a4d1f68; no voters:
I20240501 08:00:03.178056 7211 leader_election.cc:288] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:00:03.178161 7216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:03.178378 7216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [term 1 LEADER]: Becoming Leader. State: Replica: 6aa238de40d94e5f9898ac424a4d1f68, State: Running, Role: LEADER
I20240501 08:00:03.178943 7216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa238de40d94e5f9898ac424a4d1f68" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45395 } }
I20240501 08:00:03.179128 7211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:00:03.180575 7218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aa238de40d94e5f9898ac424a4d1f68. Latest consensus state: current_term: 1 leader_uuid: "6aa238de40d94e5f9898ac424a4d1f68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa238de40d94e5f9898ac424a4d1f68" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45395 } } }
I20240501 08:00:03.180583 7217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aa238de40d94e5f9898ac424a4d1f68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa238de40d94e5f9898ac424a4d1f68" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45395 } } }
I20240501 08:00:03.180732 7218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [sys.catalog]: This master's current role is: LEADER
I20240501 08:00:03.180732 7217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68 [sys.catalog]: This master's current role is: LEADER
W20240501 08:00:03.183619 7231 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 08:00:03.183701 7231 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 08:00:03.183771 7230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:00:03.184644 7230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:00:03.189500 7230 catalog_manager.cc:1347] Generated new cluster ID: 5bae6f69fee549be8992519148207022
I20240501 08:00:03.189574 7230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:00:03.205725 7230 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:00:03.206739 7230 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:00:03.214674 7230 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6aa238de40d94e5f9898ac424a4d1f68: Generated new TSK 0
I20240501 08:00:03.215575 7230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:00:03.223742 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45395
--builtin_ntp_servers=127.0.99.148:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:00:03.337688 7235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:03.337944 7235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:00:03.338045 7235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:03.341337 7235 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:00:03.341424 7235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:00:03.341532 7235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 08:00:03.345335 7235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45395
--never_fsync=true
--heap_profile_path=/tmp/kudu.7235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 08:00:03.346355 7235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:00:03.347506 7235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:03.356714 7240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:03.357434 7241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:03.358539 7243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:03.359586 7235 server_base.cc:1034] running on GCE node
I20240501 08:00:03.360049 7235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:00:03.360648 7235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:00:03.361788 7235 hybrid_clock.cc:648] HybridClock initialized: now 1714550403361772 us; error 32 us; skew 500 ppm
I20240501 08:00:03.363922 7235 webserver.cc:460] Webserver started at http://127.0.99.129:39783/ using document root <none> and password file <none>
I20240501 08:00:03.364500 7235 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:03.364558 7235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:03.364744 7235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:00:03.366503 7235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b4f8020266b5455b872a2973fc396578"
format_stamp: "Formatted at 2024-05-01 08:00:03 on dist-test-slave-wslg"
I20240501 08:00:03.366981 7235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b4f8020266b5455b872a2973fc396578"
format_stamp: "Formatted at 2024-05-01 08:00:03 on dist-test-slave-wslg"
I20240501 08:00:03.370473 7235 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:00:03.372785 7249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:03.373855 7235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:00:03.373956 7235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b4f8020266b5455b872a2973fc396578"
format_stamp: "Formatted at 2024-05-01 08:00:03 on dist-test-slave-wslg"
I20240501 08:00:03.374081 7235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:03.383574 7235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:03.384197 7235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:00:03.384387 7235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:03.384939 7235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:03.385896 7235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:03.385946 7235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:03.386056 7235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:03.386085 7235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:03.395386 7235 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:40263
I20240501 08:00:03.395449 7362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:40263 every 8 connection(s)
I20240501 08:00:03.396443 7235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:00:03.399955 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 7235
I20240501 08:00:03.400085 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:00:03.402462 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45395
--builtin_ntp_servers=127.0.99.148:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:00:03.406659 7363 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45395
I20240501 08:00:03.406934 7363 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:03.407549 7363 heartbeater.cc:511] Master 127.0.99.190:45395 requested a full tablet report, sending...
I20240501 08:00:03.409119 7175 ts_manager.cc:194] Registered new tserver with Master: b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:03.410374 7175 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:44127
W20240501 08:00:03.513427 7366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:03.513686 7366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:00:03.513764 7366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:03.516885 7366 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:00:03.516974 7366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:00:03.517081 7366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 08:00:03.520922 7366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45395
--never_fsync=true
--heap_profile_path=/tmp/kudu.7366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 08:00:03.521935 7366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:00:03.523141 7366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:03.532908 7374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:03.533725 7372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:03.534716 7371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:03.536052 7366 server_base.cc:1034] running on GCE node
I20240501 08:00:03.536602 7366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:00:03.537262 7366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:00:03.538448 7366 hybrid_clock.cc:648] HybridClock initialized: now 1714550403538426 us; error 37 us; skew 500 ppm
I20240501 08:00:03.540693 7366 webserver.cc:460] Webserver started at http://127.0.99.130:38513/ using document root <none> and password file <none>
I20240501 08:00:03.541292 7366 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:03.541368 7366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:03.541586 7366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:00:03.543439 7366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
format_stamp: "Formatted at 2024-05-01 08:00:03 on dist-test-slave-wslg"
I20240501 08:00:03.543943 7366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
format_stamp: "Formatted at 2024-05-01 08:00:03 on dist-test-slave-wslg"
I20240501 08:00:03.547514 7366 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240501 08:00:03.549862 7380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:03.550963 7366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:00:03.551090 7366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
format_stamp: "Formatted at 2024-05-01 08:00:03 on dist-test-slave-wslg"
I20240501 08:00:03.551193 7366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:03.567799 7366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:03.568507 7366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:00:03.568681 7366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:03.569227 7366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:03.570212 7366 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:03.570448 7366 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:03.570566 7366 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:03.570600 7366 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:03.580796 7366 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:40707
I20240501 08:00:03.580850 7493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:40707 every 8 connection(s)
I20240501 08:00:03.581807 7366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:00:03.589871 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 7366
I20240501 08:00:03.590029 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:00:03.591622 7494 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45395
I20240501 08:00:03.591920 7494 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:03.592721 7494 heartbeater.cc:511] Master 127.0.99.190:45395 requested a full tablet report, sending...
I20240501 08:00:03.592725 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45395
--builtin_ntp_servers=127.0.99.148:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:00:03.594074 7174 ts_manager.cc:194] Registered new tserver with Master: d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:03.594710 7174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:52407
W20240501 08:00:03.706880 7497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:03.707142 7497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:00:03.707190 7497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:03.710433 7497 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:00:03.710518 7497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:00:03.710592 7497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 08:00:03.714654 7497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45395
--never_fsync=true
--heap_profile_path=/tmp/kudu.7497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 08:00:03.715626 7497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:00:03.716814 7497 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:03.726095 7502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:03.726922 7503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:03.728036 7505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:03.729013 7497 server_base.cc:1034] running on GCE node
I20240501 08:00:03.729501 7497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:00:03.730191 7497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:00:03.731355 7497 hybrid_clock.cc:648] HybridClock initialized: now 1714550403731344 us; error 35 us; skew 500 ppm
I20240501 08:00:03.733615 7497 webserver.cc:460] Webserver started at http://127.0.99.131:44011/ using document root <none> and password file <none>
I20240501 08:00:03.734282 7497 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:03.734355 7497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:03.734544 7497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:00:03.736326 7497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "31b02f06be8a4455b94b1254d77005bd"
format_stamp: "Formatted at 2024-05-01 08:00:03 on dist-test-slave-wslg"
I20240501 08:00:03.736829 7497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "31b02f06be8a4455b94b1254d77005bd"
format_stamp: "Formatted at 2024-05-01 08:00:03 on dist-test-slave-wslg"
I20240501 08:00:03.740578 7497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 08:00:03.742980 7511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:03.744055 7497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:00:03.744192 7497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "31b02f06be8a4455b94b1254d77005bd"
format_stamp: "Formatted at 2024-05-01 08:00:03 on dist-test-slave-wslg"
I20240501 08:00:03.744297 7497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:03.782727 7497 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:03.783413 7497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:00:03.783594 7497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:03.784171 7497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:03.785130 7497 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:03.785183 7497 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:03.785269 7497 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:03.785298 7497 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:03.795634 7497 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:37861
I20240501 08:00:03.795657 7624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:37861 every 8 connection(s)
I20240501 08:00:03.796736 7497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:00:03.799703 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 7497
I20240501 08:00:03.799844 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:00:03.802134 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45395
--builtin_ntp_servers=127.0.99.148:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:00:03.807336 7625 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45395
I20240501 08:00:03.807615 7625 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:03.808243 7625 heartbeater.cc:511] Master 127.0.99.190:45395 requested a full tablet report, sending...
I20240501 08:00:03.809456 7174 ts_manager.cc:194] Registered new tserver with Master: 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:03.810082 7174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:39829
W20240501 08:00:03.918530 7628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:03.918807 7628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:00:03.918881 7628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:03.922168 7628 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:00:03.922260 7628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:00:03.922370 7628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 08:00:03.926436 7628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45395
--never_fsync=true
--heap_profile_path=/tmp/kudu.7628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 08:00:03.927459 7628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:00:03.928627 7628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:03.938189 7634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:03.939253 7636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:03.940197 7633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:03.941488 7628 server_base.cc:1034] running on GCE node
I20240501 08:00:03.941954 7628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:00:03.942642 7628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:00:03.943818 7628 hybrid_clock.cc:648] HybridClock initialized: now 1714550403943802 us; error 32 us; skew 500 ppm
I20240501 08:00:03.946132 7628 webserver.cc:460] Webserver started at http://127.0.99.132:45591/ using document root <none> and password file <none>
I20240501 08:00:03.946744 7628 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:03.946812 7628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:03.947032 7628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:00:03.949043 7628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "09a7b6e2731a474b80411efb62323604"
format_stamp: "Formatted at 2024-05-01 08:00:03 on dist-test-slave-wslg"
I20240501 08:00:03.949587 7628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "09a7b6e2731a474b80411efb62323604"
format_stamp: "Formatted at 2024-05-01 08:00:03 on dist-test-slave-wslg"
I20240501 08:00:03.953290 7628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.003s
I20240501 08:00:03.955845 7642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:03.957015 7628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:03.957155 7628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "09a7b6e2731a474b80411efb62323604"
format_stamp: "Formatted at 2024-05-01 08:00:03 on dist-test-slave-wslg"
I20240501 08:00:03.957266 7628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:03.988881 7628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:03.989595 7628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:00:03.989779 7628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:03.990427 7628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:03.991428 7628 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:03.991487 7628 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:03.991554 7628 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:03.991591 7628 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:04.001235 7628 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:35789
I20240501 08:00:04.001294 7755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:35789 every 8 connection(s)
I20240501 08:00:04.002354 7628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:00:04.009541 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 7628
I20240501 08:00:04.009686 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:00:04.012416 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45395
--builtin_ntp_servers=127.0.99.148:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:00:04.012794 7756 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45395
I20240501 08:00:04.013038 7756 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:04.013628 7756 heartbeater.cc:511] Master 127.0.99.190:45395 requested a full tablet report, sending...
I20240501 08:00:04.014788 7174 ts_manager.cc:194] Registered new tserver with Master: 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:04.015599 7174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:42193
W20240501 08:00:04.128484 7759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:04.128751 7759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:00:04.128799 7759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:04.132055 7759 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:00:04.132145 7759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:00:04.132218 7759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 08:00:04.136193 7759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45395
--never_fsync=true
--heap_profile_path=/tmp/kudu.7759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 08:00:04.137421 7759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:00:04.138687 7759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:04.148365 7764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:04.149395 7765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:04.150346 7767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:04.151613 7759 server_base.cc:1034] running on GCE node
I20240501 08:00:04.152140 7759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:00:04.152805 7759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:00:04.154018 7759 hybrid_clock.cc:648] HybridClock initialized: now 1714550404153969 us; error 29 us; skew 500 ppm
I20240501 08:00:04.156291 7759 webserver.cc:460] Webserver started at http://127.0.99.133:39227/ using document root <none> and password file <none>
I20240501 08:00:04.156904 7759 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:04.156970 7759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:04.157179 7759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:00:04.159298 7759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d8aa5ac376a54e1f8417e5af11d28fa7"
format_stamp: "Formatted at 2024-05-01 08:00:04 on dist-test-slave-wslg"
I20240501 08:00:04.159895 7759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d8aa5ac376a54e1f8417e5af11d28fa7"
format_stamp: "Formatted at 2024-05-01 08:00:04 on dist-test-slave-wslg"
I20240501 08:00:04.164455 7759 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:00:04.167574 7773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:04.168965 7759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 08:00:04.169118 7759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d8aa5ac376a54e1f8417e5af11d28fa7"
format_stamp: "Formatted at 2024-05-01 08:00:04 on dist-test-slave-wslg"
I20240501 08:00:04.169243 7759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:04.182344 7759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:04.183004 7759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:00:04.183188 7759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:04.183727 7759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:04.184684 7759 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:04.184734 7759 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:04.184798 7759 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:04.184823 7759 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:04.193498 7759 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:33901
I20240501 08:00:04.193529 7886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:33901 every 8 connection(s)
I20240501 08:00:04.194617 7759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:00:04.199404 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 7759
I20240501 08:00:04.199548 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:00:04.202311 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45395
--builtin_ntp_servers=127.0.99.148:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:00:04.205083 7887 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45395
I20240501 08:00:04.205391 7887 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:04.206039 7887 heartbeater.cc:511] Master 127.0.99.190:45395 requested a full tablet report, sending...
I20240501 08:00:04.207222 7174 ts_manager.cc:194] Registered new tserver with Master: d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901)
I20240501 08:00:04.208123 7174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:37937
W20240501 08:00:04.319602 7890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:04.319892 7890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:00:04.319939 7890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:04.323325 7890 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:00:04.323422 7890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:00:04.323500 7890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 08:00:04.327572 7890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45395
--never_fsync=true
--heap_profile_path=/tmp/kudu.7890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 08:00:04.328641 7890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:00:04.329838 7890 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:04.339560 7896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:04.340534 7898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:04.341315 7895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:04.342609 7890 server_base.cc:1034] running on GCE node
I20240501 08:00:04.343084 7890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:00:04.343703 7890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:00:04.344861 7890 hybrid_clock.cc:648] HybridClock initialized: now 1714550404344844 us; error 32 us; skew 500 ppm
I20240501 08:00:04.347031 7890 webserver.cc:460] Webserver started at http://127.0.99.134:38491/ using document root <none> and password file <none>
I20240501 08:00:04.347635 7890 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:04.347702 7890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:04.347887 7890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:00:04.349786 7890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "3b69c5d799e14e438df760453438aa75"
format_stamp: "Formatted at 2024-05-01 08:00:04 on dist-test-slave-wslg"
I20240501 08:00:04.350327 7890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "3b69c5d799e14e438df760453438aa75"
format_stamp: "Formatted at 2024-05-01 08:00:04 on dist-test-slave-wslg"
I20240501 08:00:04.353921 7890 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:00:04.356632 7904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:04.357932 7890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 08:00:04.358124 7890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "3b69c5d799e14e438df760453438aa75"
format_stamp: "Formatted at 2024-05-01 08:00:04 on dist-test-slave-wslg"
I20240501 08:00:04.358235 7890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:04.377920 7890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:04.378710 7890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:00:04.378898 7890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:04.379616 7890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:04.380558 7890 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:04.380617 7890 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:04.380685 7890 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:04.380715 7890 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:04.389453 7890 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:33589
I20240501 08:00:04.389479 8017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:33589 every 8 connection(s)
I20240501 08:00:04.390571 7890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:00:04.399499 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 7890
I20240501 08:00:04.399637 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 08:00:04.400353 8018 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45395
I20240501 08:00:04.400628 8018 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:04.401340 8018 heartbeater.cc:511] Master 127.0.99.190:45395 requested a full tablet report, sending...
I20240501 08:00:04.402137 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45395
--builtin_ntp_servers=127.0.99.148:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:00:04.402505 7174 ts_manager.cc:194] Registered new tserver with Master: 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589)
I20240501 08:00:04.403132 7174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:34503
I20240501 08:00:04.412781 7363 heartbeater.cc:503] Master 127.0.99.190:45395 was elected leader, sending a full tablet report...
W20240501 08:00:04.521829 8021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:04.522171 8021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:00:04.522253 8021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:04.525511 8021 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:00:04.525604 8021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:00:04.525714 8021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 08:00:04.529579 8021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45395
--never_fsync=true
--heap_profile_path=/tmp/kudu.8021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 08:00:04.530645 8021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:00:04.531824 8021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:04.541790 8027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:04.542577 8026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:04.543524 8029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:04.544721 8021 server_base.cc:1034] running on GCE node
I20240501 08:00:04.545249 8021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:00:04.545897 8021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:00:04.547094 8021 hybrid_clock.cc:648] HybridClock initialized: now 1714550404547075 us; error 34 us; skew 500 ppm
I20240501 08:00:04.549443 8021 webserver.cc:460] Webserver started at http://127.0.99.135:33759/ using document root <none> and password file <none>
I20240501 08:00:04.550089 8021 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:04.550153 8021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:04.550339 8021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:00:04.552104 8021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "1c8cb9b1059a47cc822ca13a2b931776"
format_stamp: "Formatted at 2024-05-01 08:00:04 on dist-test-slave-wslg"
I20240501 08:00:04.552597 8021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "1c8cb9b1059a47cc822ca13a2b931776"
format_stamp: "Formatted at 2024-05-01 08:00:04 on dist-test-slave-wslg"
I20240501 08:00:04.556250 8021 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240501 08:00:04.558869 8035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:04.560066 8021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:04.560200 8021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "1c8cb9b1059a47cc822ca13a2b931776"
format_stamp: "Formatted at 2024-05-01 08:00:04 on dist-test-slave-wslg"
I20240501 08:00:04.560317 8021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:04.579867 8021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:04.580562 8021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:00:04.580780 8021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:04.581367 8021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:04.582315 8021 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:04.582369 8021 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:04.582415 8021 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:04.582448 8021 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:04.591315 8021 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:41471
I20240501 08:00:04.591334 8148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:41471 every 8 connection(s)
I20240501 08:00:04.592424 8021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:00:04.597420 7494 heartbeater.cc:503] Master 127.0.99.190:45395 was elected leader, sending a full tablet report...
I20240501 08:00:04.598927 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 8021
I20240501 08:00:04.599040 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 08:00:04.601320 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45395
--builtin_ntp_servers=127.0.99.148:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:00:04.602654 8149 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45395
I20240501 08:00:04.602913 8149 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:04.603533 8149 heartbeater.cc:511] Master 127.0.99.190:45395 requested a full tablet report, sending...
I20240501 08:00:04.604681 7174 ts_manager.cc:194] Registered new tserver with Master: 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471)
I20240501 08:00:04.605573 7174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:49967
W20240501 08:00:04.717025 8152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:04.717278 8152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:00:04.717324 8152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:04.720511 8152 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:00:04.720599 8152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:00:04.720670 8152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240501 08:00:04.724623 8152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45395
--never_fsync=true
--heap_profile_path=/tmp/kudu.8152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 08:00:04.725569 8152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:00:04.726791 8152 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:04.736460 8157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:04.737419 8158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:04.738438 8160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:04.739671 8152 server_base.cc:1034] running on GCE node
I20240501 08:00:04.740157 8152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:00:04.740823 8152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:00:04.742041 8152 hybrid_clock.cc:648] HybridClock initialized: now 1714550404741993 us; error 62 us; skew 500 ppm
I20240501 08:00:04.744217 8152 webserver.cc:460] Webserver started at http://127.0.99.136:42201/ using document root <none> and password file <none>
I20240501 08:00:04.744810 8152 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:04.744870 8152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:04.745034 8152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:00:04.746904 8152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "1807c0c2e7d04b20b0cda311c42eef8e"
format_stamp: "Formatted at 2024-05-01 08:00:04 on dist-test-slave-wslg"
I20240501 08:00:04.747475 8152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "1807c0c2e7d04b20b0cda311c42eef8e"
format_stamp: "Formatted at 2024-05-01 08:00:04 on dist-test-slave-wslg"
I20240501 08:00:04.751150 8152 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:00:04.753587 8166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:04.754750 8152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:00:04.754868 8152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "1807c0c2e7d04b20b0cda311c42eef8e"
format_stamp: "Formatted at 2024-05-01 08:00:04 on dist-test-slave-wslg"
I20240501 08:00:04.754972 8152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:04.774132 8152 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:04.774775 8152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:00:04.774946 8152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:04.775519 8152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:04.776541 8152 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:04.776592 8152 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:04.776640 8152 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:04.776670 8152 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:04.786113 8152 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:42587
I20240501 08:00:04.786182 8279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:42587 every 8 connection(s)
I20240501 08:00:04.787173 8152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:00:04.788058 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 8152
I20240501 08:00:04.788170 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 08:00:04.796694 8280 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45395
I20240501 08:00:04.796948 8280 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:04.797557 8280 heartbeater.cc:511] Master 127.0.99.190:45395 requested a full tablet report, sending...
I20240501 08:00:04.798803 7174 ts_manager.cc:194] Registered new tserver with Master: 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587)
I20240501 08:00:04.799451 7174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:57779
I20240501 08:00:04.804154 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 08:00:04.812260 7625 heartbeater.cc:503] Master 127.0.99.190:45395 was elected leader, sending a full tablet report...
I20240501 08:00:04.814802 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 08:00:04.814893 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 7759
I20240501 08:00:04.820612 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 7890
I20240501 08:00:04.826086 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 8021
I20240501 08:00:04.831962 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu with pid 8152
I20240501 08:00:05.017906 7756 heartbeater.cc:503] Master 127.0.99.190:45395 was elected leader, sending a full tablet report...
I20240501 08:00:08.589270 7174 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34686:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:00:08.611698 7688 tablet_service.cc:1467] Processing CreateTablet for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626 (DEFAULT_TABLE table=rebalance_test_table_0 [id=61deeb642b0b4f06a2ef4b1a391fdcb9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:08.611701 7559 tablet_service.cc:1467] Processing CreateTablet for tablet 3abb5202d8d24d9b9a67f7a87b7300a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=61deeb642b0b4f06a2ef4b1a391fdcb9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:08.612453 7558 tablet_service.cc:1467] Processing CreateTablet for tablet b357c740aca144b7b063845726f419e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=61deeb642b0b4f06a2ef4b1a391fdcb9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:08.612797 7559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3abb5202d8d24d9b9a67f7a87b7300a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:08.612797 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:08.611698 7690 tablet_service.cc:1467] Processing CreateTablet for tablet b357c740aca144b7b063845726f419e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=61deeb642b0b4f06a2ef4b1a391fdcb9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:08.613878 7427 tablet_service.cc:1467] Processing CreateTablet for tablet 3abb5202d8d24d9b9a67f7a87b7300a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=61deeb642b0b4f06a2ef4b1a391fdcb9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:08.614392 7558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b357c740aca144b7b063845726f419e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:08.614493 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b357c740aca144b7b063845726f419e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:08.614934 7427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3abb5202d8d24d9b9a67f7a87b7300a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:08.616137 7297 tablet_service.cc:1467] Processing CreateTablet for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626 (DEFAULT_TABLE table=rebalance_test_table_0 [id=61deeb642b0b4f06a2ef4b1a391fdcb9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:08.617102 7297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:08.618314 7296 tablet_service.cc:1467] Processing CreateTablet for tablet 3abb5202d8d24d9b9a67f7a87b7300a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=61deeb642b0b4f06a2ef4b1a391fdcb9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:08.618607 7296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3abb5202d8d24d9b9a67f7a87b7300a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:08.613873 7428 tablet_service.cc:1467] Processing CreateTablet for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626 (DEFAULT_TABLE table=rebalance_test_table_0 [id=61deeb642b0b4f06a2ef4b1a391fdcb9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:08.618942 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:08.622346 7295 tablet_service.cc:1467] Processing CreateTablet for tablet b357c740aca144b7b063845726f419e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=61deeb642b0b4f06a2ef4b1a391fdcb9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:08.622661 7295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b357c740aca144b7b063845726f419e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:08.624240 8302 tablet_bootstrap.cc:492] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:08.624833 8300 tablet_bootstrap.cc:492] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:08.626953 8302 tablet_bootstrap.cc:654] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.626952 8300 tablet_bootstrap.cc:654] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.627437 8303 tablet_bootstrap.cc:492] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:08.627846 8300 log.cc:824] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd: Log is configured to *not* fsync() on all Append() calls
I20240501 08:00:08.627869 8302 log.cc:824] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f: Log is configured to *not* fsync() on all Append() calls
I20240501 08:00:08.628279 8301 tablet_bootstrap.cc:492] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:08.629690 8303 tablet_bootstrap.cc:654] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.630364 8301 tablet_bootstrap.cc:654] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.630610 8300 tablet_bootstrap.cc:492] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:08.630614 8303 log.cc:824] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578: Log is configured to *not* fsync() on all Append() calls
I20240501 08:00:08.630798 8300 ts_tablet_manager.cc:1367] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 08:00:08.630870 8302 tablet_bootstrap.cc:492] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:08.631049 8302 ts_tablet_manager.cc:1367] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 08:00:08.631165 8301 log.cc:824] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604: Log is configured to *not* fsync() on all Append() calls
I20240501 08:00:08.632464 8303 tablet_bootstrap.cc:492] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:08.632640 8303 ts_tablet_manager.cc:1367] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20240501 08:00:08.632761 8301 tablet_bootstrap.cc:492] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:08.632915 8301 ts_tablet_manager.cc:1367] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:00:08.635039 8300 raft_consensus.cc:348] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.635289 8300 raft_consensus.cc:374] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.635340 8300 raft_consensus.cc:729] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.635432 8302 raft_consensus.cc:348] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.635893 8300 consensus_queue.cc:260] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.636464 8303 raft_consensus.cc:348] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.636659 8303 raft_consensus.cc:374] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.636706 8303 raft_consensus.cc:729] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.636682 8301 raft_consensus.cc:348] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.635646 8302 raft_consensus.cc:374] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.636868 8301 raft_consensus.cc:374] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.636874 8302 raft_consensus.cc:729] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.636929 8301 raft_consensus.cc:729] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.636941 8300 ts_tablet_manager.cc:1398] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 08:00:08.637172 8303 consensus_queue.cc:260] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.637341 8301 consensus_queue.cc:260] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.637341 8302 consensus_queue.cc:260] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.638224 8301 ts_tablet_manager.cc:1398] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20240501 08:00:08.638228 8302 ts_tablet_manager.cc:1398] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 08:00:08.638629 8302 tablet_bootstrap.cc:492] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:08.639549 8301 tablet_bootstrap.cc:492] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:08.639601 8303 ts_tablet_manager.cc:1398] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 08:00:08.640005 8303 tablet_bootstrap.cc:492] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:08.640983 8300 tablet_bootstrap.cc:492] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:08.641068 8303 tablet_bootstrap.cc:654] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.641502 8301 tablet_bootstrap.cc:654] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.642237 8300 tablet_bootstrap.cc:654] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.642758 8301 tablet_bootstrap.cc:492] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:08.642832 8303 tablet_bootstrap.cc:492] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:08.642846 8301 ts_tablet_manager.cc:1367] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 08:00:08.642922 8303 ts_tablet_manager.cc:1367] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:00:08.643383 8300 tablet_bootstrap.cc:492] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:08.643394 8302 tablet_bootstrap.cc:654] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.643476 8300 ts_tablet_manager.cc:1367] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:00:08.643569 8303 raft_consensus.cc:348] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.643697 8303 raft_consensus.cc:374] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.643735 8303 raft_consensus.cc:729] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.643895 8303 consensus_queue.cc:260] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.644042 8300 raft_consensus.cc:348] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:08.644160 8300 raft_consensus.cc:374] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.644199 8303 ts_tablet_manager.cc:1398] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 08:00:08.644198 8300 raft_consensus.cc:729] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.644332 8300 consensus_queue.cc:260] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:08.644377 8303 tablet_bootstrap.cc:492] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:08.644708 8300 ts_tablet_manager.cc:1398] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 08:00:08.645332 8303 tablet_bootstrap.cc:654] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.645618 8301 raft_consensus.cc:348] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.645738 8301 raft_consensus.cc:374] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.645785 8301 raft_consensus.cc:729] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.645956 8301 consensus_queue.cc:260] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.647333 8303 tablet_bootstrap.cc:492] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:08.647421 8303 ts_tablet_manager.cc:1367] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:00:08.647459 8301 ts_tablet_manager.cc:1398] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:00:08.647929 8303 raft_consensus.cc:348] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:08.648058 8303 raft_consensus.cc:374] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.648109 8303 raft_consensus.cc:729] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.648265 8303 consensus_queue.cc:260] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:08.648558 8303 ts_tablet_manager.cc:1398] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:08.648901 8302 tablet_bootstrap.cc:492] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:08.649012 8302 ts_tablet_manager.cc:1367] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 08:00:08.649660 8302 raft_consensus.cc:348] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:08.649796 8302 raft_consensus.cc:374] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.649838 8302 raft_consensus.cc:729] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.650004 8302 consensus_queue.cc:260] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:08.650343 8302 ts_tablet_manager.cc:1398] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 08:00:08.651335 7364 tablet.cc:2369] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:00:08.669387 8309 raft_consensus.cc:482] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:08.669610 8309 raft_consensus.cc:504] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.670935 8309 leader_election.cc:288] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707), 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:08.674917 7448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8fdc2b9c2ef44b0bfec59a1a2f6b626" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" is_pre_election: true
I20240501 08:00:08.674918 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8fdc2b9c2ef44b0bfec59a1a2f6b626" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604" is_pre_election: true
I20240501 08:00:08.675233 7710 raft_consensus.cc:2454] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 0.
I20240501 08:00:08.675241 7448 raft_consensus.cc:2454] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 0.
I20240501 08:00:08.675700 7250 leader_election.cc:302] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4f8020266b5455b872a2973fc396578, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:08.675976 8309 raft_consensus.cc:2781] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:08.676046 8309 raft_consensus.cc:482] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:08.676075 8309 raft_consensus.cc:3037] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.677121 8309 raft_consensus.cc:504] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.677553 8309 leader_election.cc:288] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 election: Requested vote from peers d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707), 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:08.677996 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8fdc2b9c2ef44b0bfec59a1a2f6b626" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604"
I20240501 08:00:08.678138 7710 raft_consensus.cc:3037] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.678206 7448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8fdc2b9c2ef44b0bfec59a1a2f6b626" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
I20240501 08:00:08.678359 7448 raft_consensus.cc:3037] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.679423 7448 raft_consensus.cc:2454] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 1.
I20240501 08:00:08.679445 7710 raft_consensus.cc:2454] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 1.
I20240501 08:00:08.679864 7253 leader_election.cc:302] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:08.680060 8309 raft_consensus.cc:2781] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:08.680317 8309 raft_consensus.cc:686] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Becoming Leader. State: Replica: b4f8020266b5455b872a2973fc396578, State: Running, Role: LEADER
I20240501 08:00:08.680644 8309 consensus_queue.cc:237] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.684123 7173 catalog_manager.cc:5484] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4f8020266b5455b872a2973fc396578 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } }
W20240501 08:00:08.757200 7757 tablet.cc:2369] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:00:08.785478 8310 raft_consensus.cc:482] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:08.785647 8310 raft_consensus.cc:504] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:08.786903 8310 leader_election.cc:288] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:08.790951 7317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3abb5202d8d24d9b9a67f7a87b7300a6" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578" is_pre_election: true
I20240501 08:00:08.790973 7579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3abb5202d8d24d9b9a67f7a87b7300a6" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd" is_pre_election: true
I20240501 08:00:08.791208 7317 raft_consensus.cc:2454] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 0.
I20240501 08:00:08.791283 7579 raft_consensus.cc:2454] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 0.
I20240501 08:00:08.791764 7384 leader_election.cc:302] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4f8020266b5455b872a2973fc396578, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:08.792148 8310 raft_consensus.cc:2781] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:08.792223 8310 raft_consensus.cc:482] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:08.792281 8310 raft_consensus.cc:3037] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.793447 8310 raft_consensus.cc:504] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:08.793869 8310 leader_election.cc:288] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 election: Requested vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:08.794406 7579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3abb5202d8d24d9b9a67f7a87b7300a6" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
I20240501 08:00:08.794413 7317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3abb5202d8d24d9b9a67f7a87b7300a6" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578"
I20240501 08:00:08.794557 7317 raft_consensus.cc:3037] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.794564 7579 raft_consensus.cc:3037] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.795656 7317 raft_consensus.cc:2454] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 1.
I20240501 08:00:08.795691 7579 raft_consensus.cc:2454] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 1.
I20240501 08:00:08.795990 7384 leader_election.cc:302] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4f8020266b5455b872a2973fc396578, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:08.796176 8310 raft_consensus.cc:2781] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:08.796459 8310 raft_consensus.cc:686] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Becoming Leader. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Running, Role: LEADER
I20240501 08:00:08.796993 8310 consensus_queue.cc:237] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:08.800369 7173 catalog_manager.cc:5484] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: term changed from 0 to 1, leader changed from <none> to d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } }
W20240501 08:00:08.801812 7626 tablet.cc:2369] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:00:08.837060 7495 tablet.cc:2369] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:00:08.875854 8309 raft_consensus.cc:482] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:08.875993 8309 raft_consensus.cc:504] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.876588 8309 leader_election.cc:288] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:08.877135 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b357c740aca144b7b063845726f419e2" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604" is_pre_election: true
I20240501 08:00:08.877308 7710 raft_consensus.cc:2454] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 0.
I20240501 08:00:08.877672 7253 leader_election.cc:302] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:08.877849 8309 raft_consensus.cc:2781] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:08.877930 8309 raft_consensus.cc:482] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:08.878005 8309 raft_consensus.cc:3037] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.878930 8309 raft_consensus.cc:504] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.879333 8309 leader_election.cc:288] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 election: Requested vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:08.879729 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b357c740aca144b7b063845726f419e2" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604"
I20240501 08:00:08.879858 7710 raft_consensus.cc:3037] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.880731 7579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b357c740aca144b7b063845726f419e2" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd" is_pre_election: true
I20240501 08:00:08.880785 7578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b357c740aca144b7b063845726f419e2" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
I20240501 08:00:08.880889 7579 raft_consensus.cc:2454] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 0.
I20240501 08:00:08.881016 7710 raft_consensus.cc:2454] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 1.
I20240501 08:00:08.881340 7253 leader_election.cc:302] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:08.881515 8309 raft_consensus.cc:2781] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:08.881606 8309 raft_consensus.cc:686] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Becoming Leader. State: Replica: b4f8020266b5455b872a2973fc396578, State: Running, Role: LEADER
I20240501 08:00:08.881748 8309 consensus_queue.cc:237] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:08.883512 7173 catalog_manager.cc:5484] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4f8020266b5455b872a2973fc396578 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } }
W20240501 08:00:09.009348 8328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:09.009635 8328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:09.012887 8328 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 08:00:09.098304 8309 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:09.107097 8321 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:09.117432 7578 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:09.118096 8310 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:09.118230 7317 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:09.118906 8310 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:09.119942 7579 raft_consensus.cc:3037] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.121037 7579 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:09.123402 8309 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:09.133283 7709 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:09.135010 8309 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:09.136981 8356 mvcc.cc:205] Tried to move back new op lower bound from 7022798475786395648 to 7022798474435346432. Current Snapshot: MvccSnapshot[applied={T|T < 7022798475737690112}]
I20240501 08:00:09.139020 8357 mvcc.cc:205] Tried to move back new op lower bound from 7022798475736879104 to 7022798474781163520. Current Snapshot: MvccSnapshot[applied={T|T < 7022798475736879104}]
I20240501 08:00:09.146509 8373 mvcc.cc:205] Tried to move back new op lower bound from 7022798475736879104 to 7022798474781163520. Current Snapshot: MvccSnapshot[applied={T|T < 7022798475736879104}]
I20240501 08:00:09.155189 8362 mvcc.cc:205] Tried to move back new op lower bound from 7022798475737690112 to 7022798474435346432. Current Snapshot: MvccSnapshot[applied={T|T < 7022798475737690112}]
INSERT report
rows total: 2000
time total: 121.145 ms
time per row: 0.0605726 ms
I20240501 08:00:09.184965 7172 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34686:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:00:09.192142 7688 tablet_service.cc:1467] Processing CreateTablet for tablet 54ced80b9ff943ec88c27b7842500b3d (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba8a985906c04117a440386d3011b936]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.192127 7427 tablet_service.cc:1467] Processing CreateTablet for tablet 54ced80b9ff943ec88c27b7842500b3d (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba8a985906c04117a440386d3011b936]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.192452 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54ced80b9ff943ec88c27b7842500b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.192452 7427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54ced80b9ff943ec88c27b7842500b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.192497 7559 tablet_service.cc:1467] Processing CreateTablet for tablet 54ced80b9ff943ec88c27b7842500b3d (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba8a985906c04117a440386d3011b936]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.192677 7426 tablet_service.cc:1467] Processing CreateTablet for tablet bde42e485c4b49ce8acd694a25ba683a (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba8a985906c04117a440386d3011b936]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.192780 7559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54ced80b9ff943ec88c27b7842500b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.192917 7426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde42e485c4b49ce8acd694a25ba683a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.194264 7558 tablet_service.cc:1467] Processing CreateTablet for tablet 21d76484f5dd499fb6fc3d534bffb779 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba8a985906c04117a440386d3011b936]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.194512 7558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d76484f5dd499fb6fc3d534bffb779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.194897 7428 tablet_service.cc:1467] Processing CreateTablet for tablet 21d76484f5dd499fb6fc3d534bffb779 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba8a985906c04117a440386d3011b936]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.194897 7690 tablet_service.cc:1467] Processing CreateTablet for tablet 21d76484f5dd499fb6fc3d534bffb779 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba8a985906c04117a440386d3011b936]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.195112 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d76484f5dd499fb6fc3d534bffb779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.195120 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d76484f5dd499fb6fc3d534bffb779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.195282 7557 tablet_service.cc:1467] Processing CreateTablet for tablet bde42e485c4b49ce8acd694a25ba683a (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba8a985906c04117a440386d3011b936]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.195508 7557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde42e485c4b49ce8acd694a25ba683a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.197067 8384 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:09.197167 7296 tablet_service.cc:1467] Processing CreateTablet for tablet bde42e485c4b49ce8acd694a25ba683a (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba8a985906c04117a440386d3011b936]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.197479 7296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde42e485c4b49ce8acd694a25ba683a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.198671 8384 tablet_bootstrap.cc:654] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.199620 8386 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:09.199754 8382 tablet_bootstrap.cc:492] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:09.200088 8384 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:09.200176 8384 ts_tablet_manager.cc:1367] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:00:09.200624 8384 raft_consensus.cc:348] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.200717 8382 tablet_bootstrap.cc:654] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.200744 8384 raft_consensus.cc:374] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.200784 8384 raft_consensus.cc:729] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.200923 8384 consensus_queue.cc:260] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.201143 8383 tablet_bootstrap.cc:492] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:09.201287 8384 ts_tablet_manager.cc:1398] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:09.201721 8386 tablet_bootstrap.cc:654] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.202159 8383 tablet_bootstrap.cc:654] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.202256 8384 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:09.202878 8386 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:09.202967 8386 ts_tablet_manager.cc:1367] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:00:09.203168 8384 tablet_bootstrap.cc:654] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.203433 8386 raft_consensus.cc:348] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.203518 8383 tablet_bootstrap.cc:492] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:09.203557 8386 raft_consensus.cc:374] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.203598 8383 ts_tablet_manager.cc:1367] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:00:09.203598 8386 raft_consensus.cc:729] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.204049 8382 tablet_bootstrap.cc:492] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:09.204056 8383 raft_consensus.cc:348] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:09.204140 8382 ts_tablet_manager.cc:1367] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:00:09.204197 8383 raft_consensus.cc:374] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.204238 8383 raft_consensus.cc:729] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.204380 8383 consensus_queue.cc:260] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:09.204658 8383 ts_tablet_manager.cc:1398] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:09.204643 8382 raft_consensus.cc:348] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:09.204752 8382 raft_consensus.cc:374] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.204787 8382 raft_consensus.cc:729] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.204829 8383 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:09.204921 8382 consensus_queue.cc:260] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:09.205284 8382 ts_tablet_manager.cc:1398] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:09.205442 8382 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:09.206212 8382 tablet_bootstrap.cc:654] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.206223 8383 tablet_bootstrap.cc:654] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.206763 8386 consensus_queue.cc:260] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.207183 8386 ts_tablet_manager.cc:1398] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:00:09.207952 8384 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:09.208038 8384 ts_tablet_manager.cc:1367] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:00:09.208535 8384 raft_consensus.cc:348] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.208659 8384 raft_consensus.cc:374] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.208698 8384 raft_consensus.cc:729] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.208726 8382 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:09.208809 8382 ts_tablet_manager.cc:1367] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:00:09.208818 8384 consensus_queue.cc:260] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.209125 8384 ts_tablet_manager.cc:1398] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:09.209215 8382 raft_consensus.cc:348] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.209306 8382 raft_consensus.cc:374] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.209321 8384 tablet_bootstrap.cc:492] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:09.209337 8382 raft_consensus.cc:729] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.209460 8382 consensus_queue.cc:260] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.209735 8382 ts_tablet_manager.cc:1398] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:09.210297 8384 tablet_bootstrap.cc:654] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.210420 8383 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:09.210510 8383 ts_tablet_manager.cc:1367] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:00:09.210984 8383 raft_consensus.cc:348] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.211087 8383 raft_consensus.cc:374] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.211119 8383 raft_consensus.cc:729] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.211232 8383 consensus_queue.cc:260] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.211503 8383 ts_tablet_manager.cc:1398] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:09.211661 8383 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:09.212616 8383 tablet_bootstrap.cc:654] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.213361 8384 tablet_bootstrap.cc:492] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:09.213490 8384 ts_tablet_manager.cc:1367] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:00:09.213804 8383 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:09.213917 8383 ts_tablet_manager.cc:1367] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:09.214074 8384 raft_consensus.cc:348] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:09.214198 8384 raft_consensus.cc:374] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.214236 8384 raft_consensus.cc:729] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.214361 8384 consensus_queue.cc:260] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:09.214406 8383 raft_consensus.cc:348] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.214498 8383 raft_consensus.cc:374] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.214537 8383 raft_consensus.cc:729] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.214664 8383 consensus_queue.cc:260] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.214905 8383 ts_tablet_manager.cc:1398] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:09.214942 8384 ts_tablet_manager.cc:1398] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:09.253562 8310 raft_consensus.cc:482] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:09.253713 8310 raft_consensus.cc:504] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:09.254299 8310 leader_election.cc:288] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:09.254789 7578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ced80b9ff943ec88c27b7842500b3d" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd" is_pre_election: true
I20240501 08:00:09.254940 7578 raft_consensus.cc:2454] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 0.
I20240501 08:00:09.255287 7381 leader_election.cc:302] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:09.255448 8310 raft_consensus.cc:2781] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:09.255502 8310 raft_consensus.cc:482] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:09.255532 8310 raft_consensus.cc:3037] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.256426 8310 raft_consensus.cc:504] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:09.256764 8310 leader_election.cc:288] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 election: Requested vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:09.257136 7578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ced80b9ff943ec88c27b7842500b3d" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
I20240501 08:00:09.257249 7578 raft_consensus.cc:3037] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.258347 7578 raft_consensus.cc:2454] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 1.
I20240501 08:00:09.258348 7709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ced80b9ff943ec88c27b7842500b3d" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604" is_pre_election: true
I20240501 08:00:09.258528 7709 raft_consensus.cc:2454] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 0.
I20240501 08:00:09.258358 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ced80b9ff943ec88c27b7842500b3d" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604"
I20240501 08:00:09.258632 7381 leader_election.cc:302] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:09.258700 7710 raft_consensus.cc:3037] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.258893 8310 raft_consensus.cc:2781] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:09.258975 8310 raft_consensus.cc:686] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Becoming Leader. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Running, Role: LEADER
I20240501 08:00:09.259119 8310 consensus_queue.cc:237] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:09.259649 7710 raft_consensus.cc:2454] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 1.
I20240501 08:00:09.260941 7173 catalog_manager.cc:5484] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: term changed from 0 to 1, leader changed from <none> to d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:09.280386 8315 raft_consensus.cc:482] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:09.280530 8315 raft_consensus.cc:504] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.281635 8315 leader_election.cc:288] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:09.285365 7448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde42e485c4b49ce8acd694a25ba683a" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" is_pre_election: true
I20240501 08:00:09.285365 7317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde42e485c4b49ce8acd694a25ba683a" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578" is_pre_election: true
I20240501 08:00:09.285519 7448 raft_consensus.cc:2454] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 0.
I20240501 08:00:09.285527 7317 raft_consensus.cc:2454] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 0.
I20240501 08:00:09.285887 7515 leader_election.cc:302] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:09.286182 8315 raft_consensus.cc:2781] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:09.286245 8315 raft_consensus.cc:482] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:09.286281 8315 raft_consensus.cc:3037] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.287102 8315 raft_consensus.cc:504] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.287449 8315 leader_election.cc:288] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 election: Requested vote from peers b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:09.287786 7317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde42e485c4b49ce8acd694a25ba683a" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578"
I20240501 08:00:09.287842 7448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde42e485c4b49ce8acd694a25ba683a" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
I20240501 08:00:09.287917 7317 raft_consensus.cc:3037] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.287947 7448 raft_consensus.cc:3037] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.288879 7317 raft_consensus.cc:2454] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 1.
I20240501 08:00:09.288880 7448 raft_consensus.cc:2454] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 1.
I20240501 08:00:09.289183 7512 leader_election.cc:302] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:09.289336 8315 raft_consensus.cc:2781] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:09.289564 8315 raft_consensus.cc:686] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Becoming Leader. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Running, Role: LEADER
I20240501 08:00:09.289848 8315 consensus_queue.cc:237] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.292320 7173 catalog_manager.cc:5484] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 31b02f06be8a4455b94b1254d77005bd (127.0.99.131). New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:09.359998 8315 raft_consensus.cc:482] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:09.360145 8315 raft_consensus.cc:504] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.360674 8315 leader_election.cc:288] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:09.361241 7448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d76484f5dd499fb6fc3d534bffb779" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" is_pre_election: true
I20240501 08:00:09.361415 7448 raft_consensus.cc:2454] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 0.
I20240501 08:00:09.361708 7512 leader_election.cc:302] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:09.361913 8315 raft_consensus.cc:2781] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:09.361961 8315 raft_consensus.cc:482] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:09.362025 8315 raft_consensus.cc:3037] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.362920 8315 raft_consensus.cc:504] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.363356 8315 leader_election.cc:288] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 election: Requested vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:09.363745 7448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d76484f5dd499fb6fc3d534bffb779" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
I20240501 08:00:09.363881 7448 raft_consensus.cc:3037] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.364528 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d76484f5dd499fb6fc3d534bffb779" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604" is_pre_election: true
I20240501 08:00:09.364553 7709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d76484f5dd499fb6fc3d534bffb779" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604"
I20240501 08:00:09.364696 7710 raft_consensus.cc:2454] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 0.
I20240501 08:00:09.364753 7448 raft_consensus.cc:2454] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 1.
I20240501 08:00:09.365042 7512 leader_election.cc:302] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:09.365232 8315 raft_consensus.cc:2781] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:09.365339 8315 raft_consensus.cc:686] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Becoming Leader. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Running, Role: LEADER
I20240501 08:00:09.365474 8315 consensus_queue.cc:237] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.367182 7173 catalog_manager.cc:5484] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 31b02f06be8a4455b94b1254d77005bd (127.0.99.131). New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } }
W20240501 08:00:09.504930 8402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:09.505192 8402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:09.508436 8402 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 08:00:09.603000 7578 raft_consensus.cc:1261] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:09.603462 7710 raft_consensus.cc:1261] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:09.604064 8325 consensus_queue.cc:1035] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:09.604955 8325 consensus_queue.cc:1035] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:09.616803 7710 raft_consensus.cc:3037] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.616866 7448 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:09.617935 7710 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:09.618482 8315 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:09.619200 8399 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:09.622830 7447 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:09.623245 7317 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:09.623726 8315 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:09.624045 8399 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 122.594 ms
time per row: 0.0612972 ms
I20240501 08:00:09.676906 7175 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34686:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:00:09.683342 7296 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d578050a40e0474484896c98e4da9f5a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.683476 7428 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d578050a40e0474484896c98e4da9f5a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.683696 7296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.683732 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.684453 7426 tablet_service.cc:1467] Processing CreateTablet for tablet 5970772c02d04e0b95a2f2feb7dcd655 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d578050a40e0474484896c98e4da9f5a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.684445 7559 tablet_service.cc:1467] Processing CreateTablet for tablet cc6f7dcc930e4937865d37cb3c3bb3d2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d578050a40e0474484896c98e4da9f5a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.684703 7426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5970772c02d04e0b95a2f2feb7dcd655. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.684705 7559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc6f7dcc930e4937865d37cb3c3bb3d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.686236 8383 tablet_bootstrap.cc:492] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:09.686246 7690 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d578050a40e0474484896c98e4da9f5a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.686620 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.686801 8384 tablet_bootstrap.cc:492] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:09.687199 8383 tablet_bootstrap.cc:654] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.687603 8384 tablet_bootstrap.cc:654] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.688527 8383 tablet_bootstrap.cc:492] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:09.688624 8383 ts_tablet_manager.cc:1367] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:09.689009 8384 tablet_bootstrap.cc:492] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:09.689072 8386 tablet_bootstrap.cc:492] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:09.689100 8384 ts_tablet_manager.cc:1367] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:09.689111 8383 raft_consensus.cc:348] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.689218 8383 raft_consensus.cc:374] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.689254 8383 raft_consensus.cc:729] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.689389 8383 consensus_queue.cc:260] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.689544 8384 raft_consensus.cc:348] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:09.689654 8383 ts_tablet_manager.cc:1398] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:09.689656 8384 raft_consensus.cc:374] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.689697 8384 raft_consensus.cc:729] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.689822 8383 tablet_bootstrap.cc:492] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:09.689826 8384 consensus_queue.cc:260] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:09.690027 8386 tablet_bootstrap.cc:654] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.690150 8384 ts_tablet_manager.cc:1398] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:09.690171 8382 tablet_bootstrap.cc:492] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:09.690788 8383 tablet_bootstrap.cc:654] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.690997 8386 tablet_bootstrap.cc:492] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:09.691088 8386 ts_tablet_manager.cc:1367] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:00:09.691251 8382 tablet_bootstrap.cc:654] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.691460 8386 raft_consensus.cc:348] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.691562 8386 raft_consensus.cc:374] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.691603 8386 raft_consensus.cc:729] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.691736 8386 consensus_queue.cc:260] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.692028 8386 ts_tablet_manager.cc:1398] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:09.692329 8382 tablet_bootstrap.cc:492] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:09.692414 8382 ts_tablet_manager.cc:1367] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:09.692735 8383 tablet_bootstrap.cc:492] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:09.692812 8383 ts_tablet_manager.cc:1367] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:00:09.692848 8382 raft_consensus.cc:348] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.692950 8382 raft_consensus.cc:374] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.692979 8382 raft_consensus.cc:729] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.693099 8382 consensus_queue.cc:260] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.693229 8383 raft_consensus.cc:348] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.693312 8383 raft_consensus.cc:374] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.693342 8383 raft_consensus.cc:729] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.693377 8382 ts_tablet_manager.cc:1398] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:09.693475 8383 consensus_queue.cc:260] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.693749 8383 ts_tablet_manager.cc:1398] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:09.694226 7295 tablet_service.cc:1467] Processing CreateTablet for tablet 5970772c02d04e0b95a2f2feb7dcd655 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d578050a40e0474484896c98e4da9f5a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.694479 7295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5970772c02d04e0b95a2f2feb7dcd655. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.696636 8386 tablet_bootstrap.cc:492] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:09.697456 8386 tablet_bootstrap.cc:654] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.693912 7296 tablet_service.cc:1467] Processing CreateTablet for tablet cc6f7dcc930e4937865d37cb3c3bb3d2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d578050a40e0474484896c98e4da9f5a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.697777 7296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc6f7dcc930e4937865d37cb3c3bb3d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.699015 8386 tablet_bootstrap.cc:492] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:09.699092 8386 ts_tablet_manager.cc:1367] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:00:09.699585 8386 raft_consensus.cc:348] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.699688 8386 raft_consensus.cc:374] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.699719 8386 raft_consensus.cc:729] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.699844 8386 consensus_queue.cc:260] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.700121 8386 ts_tablet_manager.cc:1398] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 08:00:09.700286 8386 tablet_bootstrap.cc:492] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:09.700628 7690 tablet_service.cc:1467] Processing CreateTablet for tablet cc6f7dcc930e4937865d37cb3c3bb3d2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d578050a40e0474484896c98e4da9f5a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.700894 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc6f7dcc930e4937865d37cb3c3bb3d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.700628 7688 tablet_service.cc:1467] Processing CreateTablet for tablet 5970772c02d04e0b95a2f2feb7dcd655 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d578050a40e0474484896c98e4da9f5a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:09.701174 8386 tablet_bootstrap.cc:654] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.701256 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5970772c02d04e0b95a2f2feb7dcd655. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.702855 8386 tablet_bootstrap.cc:492] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:09.702944 8386 ts_tablet_manager.cc:1367] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:00:09.703088 8382 tablet_bootstrap.cc:492] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:09.703459 8386 raft_consensus.cc:348] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:09.703569 8386 raft_consensus.cc:374] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.703606 8386 raft_consensus.cc:729] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.703729 8386 consensus_queue.cc:260] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:09.703840 8382 tablet_bootstrap.cc:654] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.704007 8386 ts_tablet_manager.cc:1398] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:09.704903 8382 tablet_bootstrap.cc:492] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:09.704995 8382 ts_tablet_manager.cc:1367] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:09.705597 8382 raft_consensus.cc:348] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.705695 8382 raft_consensus.cc:374] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.705719 8382 raft_consensus.cc:729] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.705850 8382 consensus_queue.cc:260] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.706221 8382 ts_tablet_manager.cc:1398] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:09.706823 8382 tablet_bootstrap.cc:492] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:09.708127 8382 tablet_bootstrap.cc:654] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:09.709391 8382 tablet_bootstrap.cc:492] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:09.709487 8382 ts_tablet_manager.cc:1367] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:00:09.709889 8382 raft_consensus.cc:348] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:09.710045 8382 raft_consensus.cc:374] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:09.710106 8382 raft_consensus.cc:729] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.710240 8382 consensus_queue.cc:260] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:09.710498 8382 ts_tablet_manager.cc:1398] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:09.724659 8309 raft_consensus.cc:482] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:09.724802 8309 raft_consensus.cc:504] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.725286 8309 leader_election.cc:288] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:09.725673 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5970772c02d04e0b95a2f2feb7dcd655" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604" is_pre_election: true
I20240501 08:00:09.725720 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5970772c02d04e0b95a2f2feb7dcd655" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" is_pre_election: true
I20240501 08:00:09.725804 7710 raft_consensus.cc:2454] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 0.
I20240501 08:00:09.725888 7447 raft_consensus.cc:2454] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 0.
I20240501 08:00:09.726145 7253 leader_election.cc:302] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:09.726318 8309 raft_consensus.cc:2781] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:09.726379 8309 raft_consensus.cc:482] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:09.726408 8309 raft_consensus.cc:3037] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.727208 8309 raft_consensus.cc:504] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.727540 8309 leader_election.cc:288] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 election: Requested vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:09.727890 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5970772c02d04e0b95a2f2feb7dcd655" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
I20240501 08:00:09.728006 7447 raft_consensus.cc:3037] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.727991 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5970772c02d04e0b95a2f2feb7dcd655" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604"
I20240501 08:00:09.728132 7710 raft_consensus.cc:3037] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.728855 7447 raft_consensus.cc:2454] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 1.
I20240501 08:00:09.728873 7710 raft_consensus.cc:2454] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 1.
I20240501 08:00:09.729140 7250 leader_election.cc:302] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4f8020266b5455b872a2973fc396578, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:09.729298 8309 raft_consensus.cc:2781] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:09.729435 8309 raft_consensus.cc:686] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Becoming Leader. State: Replica: b4f8020266b5455b872a2973fc396578, State: Running, Role: LEADER
I20240501 08:00:09.729599 8309 consensus_queue.cc:237] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.731288 7167 catalog_manager.cc:5484] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4f8020266b5455b872a2973fc396578 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:09.793542 8309 raft_consensus.cc:482] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:09.793704 8309 raft_consensus.cc:504] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.794342 8309 leader_election.cc:288] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:09.794804 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ae3c82c9d4c5c858fa3f9d5959eb7" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" is_pre_election: true
I20240501 08:00:09.794804 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ae3c82c9d4c5c858fa3f9d5959eb7" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604" is_pre_election: true
I20240501 08:00:09.794950 7447 raft_consensus.cc:2454] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 0.
I20240501 08:00:09.794955 7710 raft_consensus.cc:2454] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 0.
I20240501 08:00:09.795308 7250 leader_election.cc:302] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4f8020266b5455b872a2973fc396578, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:09.795480 8309 raft_consensus.cc:2781] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:09.795662 8309 raft_consensus.cc:482] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:09.795722 8309 raft_consensus.cc:3037] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.796619 8309 raft_consensus.cc:504] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.796968 8309 leader_election.cc:288] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 election: Requested vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:09.797428 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ae3c82c9d4c5c858fa3f9d5959eb7" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604"
I20240501 08:00:09.797483 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ae3c82c9d4c5c858fa3f9d5959eb7" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
I20240501 08:00:09.797564 7710 raft_consensus.cc:3037] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.797607 7447 raft_consensus.cc:3037] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.798429 7710 raft_consensus.cc:2454] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 1.
I20240501 08:00:09.798573 7447 raft_consensus.cc:2454] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 1.
I20240501 08:00:09.798789 7253 leader_election.cc:302] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:09.799176 8309 raft_consensus.cc:2781] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:09.799260 8309 raft_consensus.cc:686] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Becoming Leader. State: Replica: b4f8020266b5455b872a2973fc396578, State: Running, Role: LEADER
I20240501 08:00:09.799399 8309 consensus_queue.cc:237] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:09.801177 7167 catalog_manager.cc:5484] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4f8020266b5455b872a2973fc396578 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:09.840731 8437 raft_consensus.cc:482] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:09.840880 8437 raft_consensus.cc:504] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:09.841358 8437 leader_election.cc:288] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263), 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:09.841889 7317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6f7dcc930e4937865d37cb3c3bb3d2" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578" is_pre_election: true
I20240501 08:00:09.841950 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6f7dcc930e4937865d37cb3c3bb3d2" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604" is_pre_election: true
I20240501 08:00:09.842073 7317 raft_consensus.cc:2454] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 0.
I20240501 08:00:09.842151 7710 raft_consensus.cc:2454] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 0.
I20240501 08:00:09.842428 7515 leader_election.cc:302] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:09.842617 8437 raft_consensus.cc:2781] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:09.842679 8437 raft_consensus.cc:482] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:09.842726 8437 raft_consensus.cc:3037] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.843564 8437 raft_consensus.cc:504] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:09.843907 8437 leader_election.cc:288] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 election: Requested vote from peers b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263), 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:09.844278 7317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6f7dcc930e4937865d37cb3c3bb3d2" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578"
I20240501 08:00:09.844348 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6f7dcc930e4937865d37cb3c3bb3d2" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604"
I20240501 08:00:09.844413 7317 raft_consensus.cc:3037] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.844466 7710 raft_consensus.cc:3037] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:09.845412 7710 raft_consensus.cc:2454] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 1.
I20240501 08:00:09.845412 7317 raft_consensus.cc:2454] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 1.
I20240501 08:00:09.845718 7515 leader_election.cc:302] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:09.845891 8437 raft_consensus.cc:2781] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:09.846017 8437 raft_consensus.cc:686] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Becoming Leader. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Running, Role: LEADER
I20240501 08:00:09.846192 8437 consensus_queue.cc:237] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:09.848088 7167 catalog_manager.cc:5484] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 31b02f06be8a4455b94b1254d77005bd (127.0.99.131). New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } }
W20240501 08:00:09.965196 8451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:09.965471 8451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:09.968664 8451 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 08:00:10.058760 7710 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:10.059346 8437 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:10.060204 7317 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:10.060863 8399 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:10.061144 7447 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:10.061563 7709 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:10.061625 8447 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:10.063350 8309 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:10.066129 7448 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:10.068884 8447 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:10.075764 7708 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:10.076217 8447 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 116.256 ms
time per row: 0.0581281 ms
I20240501 08:00:10.129623 7173 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34686:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:00:10.135778 7559 tablet_service.cc:1467] Processing CreateTablet for tablet 1088a116ccec4020aec562a69791a188 (DEFAULT_TABLE table=rebalance_test_table_3 [id=013d2bd412784602ac81aa27c405ec0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.135857 7426 tablet_service.cc:1467] Processing CreateTablet for tablet 1088a116ccec4020aec562a69791a188 (DEFAULT_TABLE table=rebalance_test_table_3 [id=013d2bd412784602ac81aa27c405ec0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.136111 7426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1088a116ccec4020aec562a69791a188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.136116 7559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1088a116ccec4020aec562a69791a188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.136346 7296 tablet_service.cc:1467] Processing CreateTablet for tablet 1088a116ccec4020aec562a69791a188 (DEFAULT_TABLE table=rebalance_test_table_3 [id=013d2bd412784602ac81aa27c405ec0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.136605 7296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1088a116ccec4020aec562a69791a188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.138312 8383 tablet_bootstrap.cc:492] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:10.138756 8386 tablet_bootstrap.cc:492] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:10.139209 8383 tablet_bootstrap.cc:654] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.139595 8386 tablet_bootstrap.cc:654] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.139600 7426 tablet_service.cc:1467] Processing CreateTablet for tablet 5a37e631ae4f46a2be1ad5c43db5724b (DEFAULT_TABLE table=rebalance_test_table_3 [id=013d2bd412784602ac81aa27c405ec0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.139807 8384 tablet_bootstrap.cc:492] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:10.139834 7426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a37e631ae4f46a2be1ad5c43db5724b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.140638 8386 tablet_bootstrap.cc:492] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:10.140645 8384 tablet_bootstrap.cc:654] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.140722 8386 ts_tablet_manager.cc:1367] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:00:10.141117 8386 raft_consensus.cc:348] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.141227 8386 raft_consensus.cc:374] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.141259 8386 raft_consensus.cc:729] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.141412 8386 consensus_queue.cc:260] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.141670 8386 ts_tablet_manager.cc:1398] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:10.141741 7559 tablet_service.cc:1467] Processing CreateTablet for tablet 0e42ef8df4884bc9bc33f5ed4c164c8c (DEFAULT_TABLE table=rebalance_test_table_3 [id=013d2bd412784602ac81aa27c405ec0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.141964 7559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e42ef8df4884bc9bc33f5ed4c164c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.142035 7557 tablet_service.cc:1467] Processing CreateTablet for tablet 5a37e631ae4f46a2be1ad5c43db5724b (DEFAULT_TABLE table=rebalance_test_table_3 [id=013d2bd412784602ac81aa27c405ec0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.142261 7557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a37e631ae4f46a2be1ad5c43db5724b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.145229 7296 tablet_service.cc:1467] Processing CreateTablet for tablet 0e42ef8df4884bc9bc33f5ed4c164c8c (DEFAULT_TABLE table=rebalance_test_table_3 [id=013d2bd412784602ac81aa27c405ec0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.145299 7690 tablet_service.cc:1467] Processing CreateTablet for tablet 5a37e631ae4f46a2be1ad5c43db5724b (DEFAULT_TABLE table=rebalance_test_table_3 [id=013d2bd412784602ac81aa27c405ec0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.145299 7688 tablet_service.cc:1467] Processing CreateTablet for tablet 0e42ef8df4884bc9bc33f5ed4c164c8c (DEFAULT_TABLE table=rebalance_test_table_3 [id=013d2bd412784602ac81aa27c405ec0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.145519 7296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e42ef8df4884bc9bc33f5ed4c164c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.145635 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a37e631ae4f46a2be1ad5c43db5724b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.145812 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e42ef8df4884bc9bc33f5ed4c164c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.147588 8382 tablet_bootstrap.cc:492] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:10.147712 8386 tablet_bootstrap.cc:492] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:10.148316 8384 tablet_bootstrap.cc:492] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:10.148417 8384 ts_tablet_manager.cc:1367] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 08:00:10.148538 8386 tablet_bootstrap.cc:654] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.148542 8382 tablet_bootstrap.cc:654] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.148950 8384 raft_consensus.cc:348] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.149091 8384 raft_consensus.cc:374] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.149134 8384 raft_consensus.cc:729] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.149267 8384 consensus_queue.cc:260] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.149554 8384 ts_tablet_manager.cc:1398] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 08:00:10.149749 8384 tablet_bootstrap.cc:492] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:10.149793 8382 tablet_bootstrap.cc:492] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:10.149873 8382 ts_tablet_manager.cc:1367] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:10.150430 8382 raft_consensus.cc:348] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.150543 8382 raft_consensus.cc:374] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.150585 8382 raft_consensus.cc:729] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.150724 8384 tablet_bootstrap.cc:654] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.150740 8382 consensus_queue.cc:260] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.151602 8382 ts_tablet_manager.cc:1398] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:10.151782 8382 tablet_bootstrap.cc:492] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:10.151818 8383 tablet_bootstrap.cc:492] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:10.151906 8383 ts_tablet_manager.cc:1367] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240501 08:00:10.152381 8383 raft_consensus.cc:348] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.152503 8383 raft_consensus.cc:374] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.152539 8383 raft_consensus.cc:729] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.152734 8382 tablet_bootstrap.cc:654] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.152676 8383 consensus_queue.cc:260] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.152978 8383 ts_tablet_manager.cc:1398] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:10.153178 8383 tablet_bootstrap.cc:492] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:10.155454 8384 tablet_bootstrap.cc:492] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:10.155560 8384 ts_tablet_manager.cc:1367] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:00:10.155956 8384 raft_consensus.cc:348] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.156069 8384 raft_consensus.cc:374] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.156108 8384 raft_consensus.cc:729] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.156116 8383 tablet_bootstrap.cc:654] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.156234 8384 consensus_queue.cc:260] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.156854 8384 ts_tablet_manager.cc:1398] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:10.157034 8384 tablet_bootstrap.cc:492] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:10.157948 8384 tablet_bootstrap.cc:654] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.158385 8383 tablet_bootstrap.cc:492] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:10.158478 8383 ts_tablet_manager.cc:1367] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:00:10.158926 8383 raft_consensus.cc:348] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.159037 8383 raft_consensus.cc:374] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.159042 8382 tablet_bootstrap.cc:492] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:10.159090 8383 raft_consensus.cc:729] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.159129 8382 ts_tablet_manager.cc:1367] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:00:10.159246 8386 tablet_bootstrap.cc:492] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:10.159258 8383 consensus_queue.cc:260] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.159341 8386 ts_tablet_manager.cc:1367] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240501 08:00:10.159540 8383 ts_tablet_manager.cc:1398] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:10.159574 8382 raft_consensus.cc:348] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.159668 8382 raft_consensus.cc:374] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.159724 8382 raft_consensus.cc:729] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.159788 8386 raft_consensus.cc:348] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.159886 8386 raft_consensus.cc:374] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.159847 8382 consensus_queue.cc:260] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.159915 8386 raft_consensus.cc:729] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.160038 8386 consensus_queue.cc:260] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.160140 8382 ts_tablet_manager.cc:1398] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:10.160292 8386 ts_tablet_manager.cc:1398] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:10.160321 8384 tablet_bootstrap.cc:492] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:10.160392 8384 ts_tablet_manager.cc:1367] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:00:10.160732 8384 raft_consensus.cc:348] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.160822 8384 raft_consensus.cc:374] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.160856 8384 raft_consensus.cc:729] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.160986 8384 consensus_queue.cc:260] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.161253 8384 ts_tablet_manager.cc:1398] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:10.237826 8447 raft_consensus.cc:482] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:10.237999 8447 raft_consensus.cc:504] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.238443 8447 leader_election.cc:288] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:10.238848 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e42ef8df4884bc9bc33f5ed4c164c8c" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604" is_pre_election: true
I20240501 08:00:10.238852 7578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e42ef8df4884bc9bc33f5ed4c164c8c" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd" is_pre_election: true
I20240501 08:00:10.238987 7708 raft_consensus.cc:2454] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 0.
I20240501 08:00:10.239008 7578 raft_consensus.cc:2454] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 0.
I20240501 08:00:10.239328 7253 leader_election.cc:302] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:10.239507 8447 raft_consensus.cc:2781] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:10.239568 8447 raft_consensus.cc:482] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:10.239624 8447 raft_consensus.cc:3037] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.240604 8447 raft_consensus.cc:504] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.240949 8447 leader_election.cc:288] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 election: Requested vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:10.241393 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e42ef8df4884bc9bc33f5ed4c164c8c" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604"
I20240501 08:00:10.241389 7578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e42ef8df4884bc9bc33f5ed4c164c8c" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
I20240501 08:00:10.241521 7708 raft_consensus.cc:3037] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.241521 7578 raft_consensus.cc:3037] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.242604 7708 raft_consensus.cc:2454] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 1.
I20240501 08:00:10.242604 7578 raft_consensus.cc:2454] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 1.
I20240501 08:00:10.242925 7253 leader_election.cc:302] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:10.243093 8447 raft_consensus.cc:2781] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:10.243201 8447 raft_consensus.cc:686] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Becoming Leader. State: Replica: b4f8020266b5455b872a2973fc396578, State: Running, Role: LEADER
I20240501 08:00:10.243366 8447 consensus_queue.cc:237] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.245031 7167 catalog_manager.cc:5484] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4f8020266b5455b872a2973fc396578 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } }
I20240501 08:00:10.262024 8447 raft_consensus.cc:482] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:10.262214 8447 raft_consensus.cc:504] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.262593 8447 leader_election.cc:288] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:10.263064 7578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1088a116ccec4020aec562a69791a188" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd" is_pre_election: true
I20240501 08:00:10.263095 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1088a116ccec4020aec562a69791a188" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" is_pre_election: true
I20240501 08:00:10.263229 7578 raft_consensus.cc:2454] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 0.
I20240501 08:00:10.263229 7447 raft_consensus.cc:2454] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 0.
I20240501 08:00:10.263558 7250 leader_election.cc:302] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4f8020266b5455b872a2973fc396578, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:10.263826 8447 raft_consensus.cc:2781] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:10.263895 8447 raft_consensus.cc:482] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:10.263929 8447 raft_consensus.cc:3037] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.264811 8447 raft_consensus.cc:504] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.265153 8447 leader_election.cc:288] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 election: Requested vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:10.265609 7578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1088a116ccec4020aec562a69791a188" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
I20240501 08:00:10.265729 7578 raft_consensus.cc:3037] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.265698 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1088a116ccec4020aec562a69791a188" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
I20240501 08:00:10.265837 7447 raft_consensus.cc:3037] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.266588 7578 raft_consensus.cc:2454] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 1.
I20240501 08:00:10.266700 7447 raft_consensus.cc:2454] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 1.
I20240501 08:00:10.266976 7250 leader_election.cc:302] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:10.267359 8447 raft_consensus.cc:2781] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:10.267475 8447 raft_consensus.cc:686] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Becoming Leader. State: Replica: b4f8020266b5455b872a2973fc396578, State: Running, Role: LEADER
I20240501 08:00:10.267616 8447 consensus_queue.cc:237] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.269356 7167 catalog_manager.cc:5484] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4f8020266b5455b872a2973fc396578 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:10.483431 8311 raft_consensus.cc:482] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:10.483582 8311 raft_consensus.cc:504] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.485076 8311 leader_election.cc:288] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:10.490061 7578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a37e631ae4f46a2be1ad5c43db5724b" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd" is_pre_election: true
I20240501 08:00:10.490242 7578 raft_consensus.cc:2454] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 0.
I20240501 08:00:10.490655 7643 leader_election.cc:302] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, 31b02f06be8a4455b94b1254d77005bd; no voters:
I20240501 08:00:10.491081 8311 raft_consensus.cc:2781] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:10.491154 8311 raft_consensus.cc:482] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:10.491202 8311 raft_consensus.cc:3037] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.491396 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a37e631ae4f46a2be1ad5c43db5724b" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" is_pre_election: true
I20240501 08:00:10.491530 7447 raft_consensus.cc:2454] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 0.
I20240501 08:00:10.492319 8311 raft_consensus.cc:504] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.492671 8311 leader_election.cc:288] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 election: Requested vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:10.493067 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a37e631ae4f46a2be1ad5c43db5724b" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
I20240501 08:00:10.493093 7578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a37e631ae4f46a2be1ad5c43db5724b" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
I20240501 08:00:10.493181 7447 raft_consensus.cc:3037] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.493191 7578 raft_consensus.cc:3037] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.493960 7447 raft_consensus.cc:2454] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 1.
I20240501 08:00:10.493960 7578 raft_consensus.cc:2454] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 1.
I20240501 08:00:10.494288 7643 leader_election.cc:302] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:10.494485 8311 raft_consensus.cc:2781] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:10.494665 8311 raft_consensus.cc:686] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Becoming Leader. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Running, Role: LEADER
I20240501 08:00:10.494966 8311 consensus_queue.cc:237] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.497538 7167 catalog_manager.cc:5484] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09a7b6e2731a474b80411efb62323604 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } }
W20240501 08:00:10.642787 8506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:10.643064 8506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:10.646342 8506 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 08:00:10.678650 8447 consensus_queue.cc:1035] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:10.682897 8309 consensus_queue.cc:1035] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:10.742270 7447 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:10.742748 8309 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:10.742878 7578 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:10.743496 8309 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:10.744380 7448 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:10.747296 8311 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:10.753872 7579 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:10.754318 8311 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 122.387 ms
time per row: 0.0611934 ms
I20240501 08:00:10.815229 7171 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34686:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:00:10.821022 7296 tablet_service.cc:1467] Processing CreateTablet for tablet 7377810e08104526b8b8acdf886a3368 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ae84a8bf8ac745fb982412e9b194d7fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.821115 7557 tablet_service.cc:1467] Processing CreateTablet for tablet 7377810e08104526b8b8acdf886a3368 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ae84a8bf8ac745fb982412e9b194d7fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.821331 7296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7377810e08104526b8b8acdf886a3368. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.821441 7557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7377810e08104526b8b8acdf886a3368. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.823271 8568 tablet_bootstrap.cc:492] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:10.823952 7426 tablet_service.cc:1467] Processing CreateTablet for tablet 7377810e08104526b8b8acdf886a3368 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ae84a8bf8ac745fb982412e9b194d7fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.824215 7426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7377810e08104526b8b8acdf886a3368. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.824257 8568 tablet_bootstrap.cc:654] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.824257 7557 tablet_service.cc:1467] Processing CreateTablet for tablet 4dde9710a5eb4baa881d9dd04affe184 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ae84a8bf8ac745fb982412e9b194d7fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.824513 7557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dde9710a5eb4baa881d9dd04affe184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.825394 8568 tablet_bootstrap.cc:492] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:10.825489 8568 ts_tablet_manager.cc:1367] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:10.826013 8568 raft_consensus.cc:348] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.826124 8568 raft_consensus.cc:374] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.826162 8568 raft_consensus.cc:729] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.826282 8571 tablet_bootstrap.cc:492] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:10.826303 8568 consensus_queue.cc:260] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.826460 7428 tablet_service.cc:1467] Processing CreateTablet for tablet 07967f68a412411494c6dd1f47209009 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ae84a8bf8ac745fb982412e9b194d7fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.826583 8568 ts_tablet_manager.cc:1398] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:10.826678 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07967f68a412411494c6dd1f47209009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.826753 8568 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:10.827284 8569 tablet_bootstrap.cc:492] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:10.827602 8568 tablet_bootstrap.cc:654] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.828289 8569 tablet_bootstrap.cc:654] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.828337 8571 tablet_bootstrap.cc:654] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.829252 7690 tablet_service.cc:1467] Processing CreateTablet for tablet 07967f68a412411494c6dd1f47209009 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ae84a8bf8ac745fb982412e9b194d7fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.829535 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07967f68a412411494c6dd1f47209009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.830066 8569 tablet_bootstrap.cc:492] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:10.830164 8569 ts_tablet_manager.cc:1367] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:00:10.830170 7688 tablet_service.cc:1467] Processing CreateTablet for tablet 4dde9710a5eb4baa881d9dd04affe184 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ae84a8bf8ac745fb982412e9b194d7fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.830394 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dde9710a5eb4baa881d9dd04affe184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.830652 8569 raft_consensus.cc:348] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.830760 8569 raft_consensus.cc:374] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.830788 8569 raft_consensus.cc:729] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.830921 8569 consensus_queue.cc:260] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.831205 8569 ts_tablet_manager.cc:1398] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:10.831743 8575 tablet_bootstrap.cc:492] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:10.832865 8571 tablet_bootstrap.cc:492] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:10.832965 8571 ts_tablet_manager.cc:1367] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:00:10.833011 8575 tablet_bootstrap.cc:654] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.832971 7296 tablet_service.cc:1467] Processing CreateTablet for tablet 07967f68a412411494c6dd1f47209009 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ae84a8bf8ac745fb982412e9b194d7fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.832971 7295 tablet_service.cc:1467] Processing CreateTablet for tablet 4dde9710a5eb4baa881d9dd04affe184 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ae84a8bf8ac745fb982412e9b194d7fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:10.833240 7296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07967f68a412411494c6dd1f47209009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.833388 7295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dde9710a5eb4baa881d9dd04affe184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.833483 8571 raft_consensus.cc:348] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.833604 8571 raft_consensus.cc:374] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.833639 8571 raft_consensus.cc:729] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.833822 8571 consensus_queue.cc:260] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.834115 8575 tablet_bootstrap.cc:492] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:10.834200 8575 ts_tablet_manager.cc:1367] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:00:10.834218 8571 ts_tablet_manager.cc:1398] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:10.834451 8571 tablet_bootstrap.cc:492] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:10.834645 8575 raft_consensus.cc:348] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.834735 8575 raft_consensus.cc:374] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.834766 8575 raft_consensus.cc:729] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.834897 8575 consensus_queue.cc:260] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.835145 8575 ts_tablet_manager.cc:1398] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:10.835336 8575 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:10.835345 8571 tablet_bootstrap.cc:654] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.835577 8569 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:10.836577 8569 tablet_bootstrap.cc:654] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.836608 8575 tablet_bootstrap.cc:654] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.837823 8569 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:10.837911 8569 ts_tablet_manager.cc:1367] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:10.838424 8569 raft_consensus.cc:348] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.838531 8569 raft_consensus.cc:374] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.838564 8569 raft_consensus.cc:729] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.838672 8569 consensus_queue.cc:260] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.838961 8569 ts_tablet_manager.cc:1398] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:10.839071 8575 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:10.839118 8569 tablet_bootstrap.cc:492] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:10.839169 8575 ts_tablet_manager.cc:1367] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:00:10.839623 8575 raft_consensus.cc:348] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.839735 8575 raft_consensus.cc:374] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.839869 8575 raft_consensus.cc:729] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.840014 8569 tablet_bootstrap.cc:654] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:10.840051 8575 consensus_queue.cc:260] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.840368 8575 ts_tablet_manager.cc:1398] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:10.841169 8569 tablet_bootstrap.cc:492] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:10.841259 8569 ts_tablet_manager.cc:1367] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:10.841892 8569 raft_consensus.cc:348] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.842053 8569 raft_consensus.cc:374] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.842089 8569 raft_consensus.cc:729] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.842219 8569 consensus_queue.cc:260] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.842264 8568 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:10.842370 8568 ts_tablet_manager.cc:1367] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 08:00:10.842571 8569 ts_tablet_manager.cc:1398] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 08:00:10.842690 8571 tablet_bootstrap.cc:492] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:10.842795 8571 ts_tablet_manager.cc:1367] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 08:00:10.842864 8568 raft_consensus.cc:348] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.842973 8568 raft_consensus.cc:374] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.843000 8568 raft_consensus.cc:729] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.843129 8568 consensus_queue.cc:260] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.843364 8315 raft_consensus.cc:482] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:10.843402 8568 ts_tablet_manager.cc:1398] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:10.843442 8315 raft_consensus.cc:504] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.843812 8315 leader_election.cc:288] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:10.844128 7317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7377810e08104526b8b8acdf886a3368" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578" is_pre_election: true
I20240501 08:00:10.844251 7317 raft_consensus.cc:2454] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 0.
I20240501 08:00:10.844229 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7377810e08104526b8b8acdf886a3368" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" is_pre_election: true
I20240501 08:00:10.844357 7447 raft_consensus.cc:2454] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 0.
I20240501 08:00:10.843248 8571 raft_consensus.cc:348] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.844477 8571 raft_consensus.cc:374] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:10.844522 8571 raft_consensus.cc:729] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.844524 7515 leader_election.cc:302] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:10.844660 8571 consensus_queue.cc:260] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.845019 8571 ts_tablet_manager.cc:1398] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:10.845248 8315 raft_consensus.cc:2781] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:10.845335 8315 raft_consensus.cc:482] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:10.845381 8315 raft_consensus.cc:3037] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.846446 8315 raft_consensus.cc:504] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.846861 8315 leader_election.cc:288] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 election: Requested vote from peers b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:10.847358 7317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7377810e08104526b8b8acdf886a3368" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578"
I20240501 08:00:10.847389 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7377810e08104526b8b8acdf886a3368" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
I20240501 08:00:10.847476 7317 raft_consensus.cc:3037] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.847483 7447 raft_consensus.cc:3037] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.848356 7317 raft_consensus.cc:2454] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 1.
I20240501 08:00:10.848371 7447 raft_consensus.cc:2454] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 1.
I20240501 08:00:10.848685 7515 leader_election.cc:302] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:10.849393 8315 raft_consensus.cc:2781] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:10.849602 8315 raft_consensus.cc:686] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Becoming Leader. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Running, Role: LEADER
I20240501 08:00:10.849720 8315 consensus_queue.cc:237] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.851276 7171 catalog_manager.cc:5484] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 31b02f06be8a4455b94b1254d77005bd (127.0.99.131). New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:10.878942 8560 raft_consensus.cc:482] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:10.879073 8560 raft_consensus.cc:504] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.879531 8560 leader_election.cc:288] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:10.880518 7579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dde9710a5eb4baa881d9dd04affe184" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd" is_pre_election: true
I20240501 08:00:10.880689 7579 raft_consensus.cc:2454] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 0.
I20240501 08:00:10.881024 7643 leader_election.cc:302] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, 31b02f06be8a4455b94b1254d77005bd; no voters:
I20240501 08:00:10.881206 8560 raft_consensus.cc:2781] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:10.881260 8560 raft_consensus.cc:482] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:10.881309 8560 raft_consensus.cc:3037] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.882331 8560 raft_consensus.cc:504] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.882715 8560 leader_election.cc:288] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 election: Requested vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:10.883184 7579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dde9710a5eb4baa881d9dd04affe184" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
I20240501 08:00:10.883323 7579 raft_consensus.cc:3037] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.883522 7317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dde9710a5eb4baa881d9dd04affe184" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578" is_pre_election: true
I20240501 08:00:10.883529 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dde9710a5eb4baa881d9dd04affe184" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578"
I20240501 08:00:10.883666 7317 raft_consensus.cc:2454] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 0.
I20240501 08:00:10.883939 8560 raft_consensus.cc:482] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:10.884042 8560 raft_consensus.cc:504] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.884359 8560 leader_election.cc:288] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:10.884436 7579 raft_consensus.cc:2454] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 1.
I20240501 08:00:10.884725 7643 leader_election.cc:302] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, 31b02f06be8a4455b94b1254d77005bd; no voters: b4f8020266b5455b872a2973fc396578
I20240501 08:00:10.884752 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07967f68a412411494c6dd1f47209009" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578" is_pre_election: true
I20240501 08:00:10.884778 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07967f68a412411494c6dd1f47209009" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" is_pre_election: true
I20240501 08:00:10.884862 7316 raft_consensus.cc:2454] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 0.
I20240501 08:00:10.884912 7447 raft_consensus.cc:2454] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 0.
I20240501 08:00:10.885035 8560 raft_consensus.cc:2781] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:10.885130 8560 raft_consensus.cc:686] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Becoming Leader. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Running, Role: LEADER
I20240501 08:00:10.885162 7646 leader_election.cc:302] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:10.885300 8560 consensus_queue.cc:237] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:10.885450 8311 raft_consensus.cc:2781] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:10.885509 8311 raft_consensus.cc:482] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:10.885547 8311 raft_consensus.cc:3037] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.886564 8311 raft_consensus.cc:504] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.886862 8311 leader_election.cc:288] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 election: Requested vote from peers b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:10.887172 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07967f68a412411494c6dd1f47209009" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578"
I20240501 08:00:10.887298 7316 raft_consensus.cc:3037] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.887663 7171 catalog_manager.cc:5484] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09a7b6e2731a474b80411efb62323604 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:10.888132 7316 raft_consensus.cc:2454] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 1.
I20240501 08:00:10.888291 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07967f68a412411494c6dd1f47209009" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
I20240501 08:00:10.888413 7447 raft_consensus.cc:3037] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:10.888432 7646 leader_election.cc:302] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:10.888581 8311 raft_consensus.cc:2781] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:10.888659 8311 raft_consensus.cc:686] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Becoming Leader. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Running, Role: LEADER
I20240501 08:00:10.888780 8311 consensus_queue.cc:237] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:10.889344 7447 raft_consensus.cc:2454] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 1.
I20240501 08:00:10.891381 7171 catalog_manager.cc:5484] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09a7b6e2731a474b80411efb62323604 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } }
W20240501 08:00:11.003839 8585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:11.004122 8585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:11.007395 8585 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 08:00:11.094627 7579 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:11.095074 8311 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:11.095760 7447 raft_consensus.cc:1261] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:11.096225 8560 consensus_queue.cc:1035] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:11.096374 7316 raft_consensus.cc:3037] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:11.097582 7316 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:11.096377 7317 raft_consensus.cc:1261] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:11.100433 8560 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:11.100776 8560 consensus_queue.cc:1035] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:11.101181 7317 raft_consensus.cc:1261] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:11.101548 8315 consensus_queue.cc:1035] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:11.102175 7448 raft_consensus.cc:1261] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:11.103972 8581 consensus_queue.cc:1035] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.815 ms
time per row: 0.0559076 ms
I20240501 08:00:11.163959 7166 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34686:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:00:11.169850 7428 tablet_service.cc:1467] Processing CreateTablet for tablet 81ccdd855b584d65860fd7a5777920a3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2427aea1b9a6436ca7bd1796a85634ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.169871 7557 tablet_service.cc:1467] Processing CreateTablet for tablet 81ccdd855b584d65860fd7a5777920a3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2427aea1b9a6436ca7bd1796a85634ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.170222 7557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ccdd855b584d65860fd7a5777920a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.170217 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ccdd855b584d65860fd7a5777920a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.172394 8571 tablet_bootstrap.cc:492] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:11.172616 7426 tablet_service.cc:1467] Processing CreateTablet for tablet dd17760cd5c2454db97610b2f71c0ee2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2427aea1b9a6436ca7bd1796a85634ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.172873 7426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd17760cd5c2454db97610b2f71c0ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.173249 8571 tablet_bootstrap.cc:654] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.173733 8568 tablet_bootstrap.cc:492] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:11.174299 8571 tablet_bootstrap.cc:492] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:11.174399 8571 ts_tablet_manager.cc:1367] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:11.174706 7559 tablet_service.cc:1467] Processing CreateTablet for tablet dd17760cd5c2454db97610b2f71c0ee2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2427aea1b9a6436ca7bd1796a85634ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.174897 7559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd17760cd5c2454db97610b2f71c0ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.174876 8571 raft_consensus.cc:348] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.174975 8568 tablet_bootstrap.cc:654] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.174995 8571 raft_consensus.cc:374] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.175026 8571 raft_consensus.cc:729] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.175181 8571 consensus_queue.cc:260] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.175482 8571 ts_tablet_manager.cc:1398] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:11.175699 8571 tablet_bootstrap.cc:492] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:11.176592 8571 tablet_bootstrap.cc:654] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.177004 7558 tablet_service.cc:1467] Processing CreateTablet for tablet e178f7c3c17c4e1ca3d16b6441f76e77 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2427aea1b9a6436ca7bd1796a85634ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.177265 7558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e178f7c3c17c4e1ca3d16b6441f76e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.177779 8571 tablet_bootstrap.cc:492] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:11.177871 8571 ts_tablet_manager.cc:1367] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:11.178325 8571 raft_consensus.cc:348] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:11.178426 8571 raft_consensus.cc:374] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.178521 8571 raft_consensus.cc:729] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.178696 8571 consensus_queue.cc:260] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:11.178975 8571 ts_tablet_manager.cc:1398] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:11.180124 7296 tablet_service.cc:1467] Processing CreateTablet for tablet e178f7c3c17c4e1ca3d16b6441f76e77 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2427aea1b9a6436ca7bd1796a85634ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.180119 7688 tablet_service.cc:1467] Processing CreateTablet for tablet 81ccdd855b584d65860fd7a5777920a3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2427aea1b9a6436ca7bd1796a85634ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.180171 7690 tablet_service.cc:1467] Processing CreateTablet for tablet dd17760cd5c2454db97610b2f71c0ee2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2427aea1b9a6436ca7bd1796a85634ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.180403 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ccdd855b584d65860fd7a5777920a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.180426 7296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e178f7c3c17c4e1ca3d16b6441f76e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.180570 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd17760cd5c2454db97610b2f71c0ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.180521 7426 tablet_service.cc:1467] Processing CreateTablet for tablet e178f7c3c17c4e1ca3d16b6441f76e77 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2427aea1b9a6436ca7bd1796a85634ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.180763 7426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e178f7c3c17c4e1ca3d16b6441f76e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.182662 8569 tablet_bootstrap.cc:492] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:11.182662 8575 tablet_bootstrap.cc:492] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:11.183495 8575 tablet_bootstrap.cc:654] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.183495 8569 tablet_bootstrap.cc:654] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.184280 8571 tablet_bootstrap.cc:492] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:11.184293 8568 tablet_bootstrap.cc:492] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:11.184394 8568 ts_tablet_manager.cc:1367] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 08:00:11.184805 8568 raft_consensus.cc:348] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.184901 8568 raft_consensus.cc:374] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.184923 8568 raft_consensus.cc:729] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.185037 8568 consensus_queue.cc:260] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.185386 8568 ts_tablet_manager.cc:1398] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:11.185421 8571 tablet_bootstrap.cc:654] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.185532 8568 tablet_bootstrap.cc:492] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:11.186491 8568 tablet_bootstrap.cc:654] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.187747 8568 tablet_bootstrap.cc:492] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:11.187764 8571 tablet_bootstrap.cc:492] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:11.187834 8568 ts_tablet_manager.cc:1367] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:00:11.187839 8571 ts_tablet_manager.cc:1367] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:00:11.188195 8568 raft_consensus.cc:348] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:11.188285 8568 raft_consensus.cc:374] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.188318 8568 raft_consensus.cc:729] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.188310 8571 raft_consensus.cc:348] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.188439 8571 raft_consensus.cc:374] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.188477 8571 raft_consensus.cc:729] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.188458 8568 consensus_queue.cc:260] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:11.188606 8571 consensus_queue.cc:260] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.188810 8568 ts_tablet_manager.cc:1398] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:11.189349 8569 tablet_bootstrap.cc:492] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:11.189383 8571 ts_tablet_manager.cc:1398] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:11.189450 8569 ts_tablet_manager.cc:1367] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 08:00:11.189831 8569 raft_consensus.cc:348] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.189924 8569 raft_consensus.cc:374] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.189960 8569 raft_consensus.cc:729] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.190114 8569 consensus_queue.cc:260] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.190374 8569 ts_tablet_manager.cc:1398] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:11.190618 8568 tablet_bootstrap.cc:492] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:11.191365 8575 tablet_bootstrap.cc:492] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:11.191460 8575 ts_tablet_manager.cc:1367] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 08:00:11.191543 8568 tablet_bootstrap.cc:654] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.191895 8575 raft_consensus.cc:348] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.192003 8575 raft_consensus.cc:374] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.192039 8575 raft_consensus.cc:729] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.192162 8575 consensus_queue.cc:260] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.192469 8575 ts_tablet_manager.cc:1398] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:11.192643 8575 tablet_bootstrap.cc:492] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:11.192734 8568 tablet_bootstrap.cc:492] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:11.192813 8568 ts_tablet_manager.cc:1367] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:00:11.193223 8568 raft_consensus.cc:348] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.193324 8568 raft_consensus.cc:374] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.193356 8568 raft_consensus.cc:729] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.193444 8575 tablet_bootstrap.cc:654] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.193493 8568 consensus_queue.cc:260] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.193794 8568 ts_tablet_manager.cc:1398] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:11.194756 8575 tablet_bootstrap.cc:492] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:11.194839 8575 ts_tablet_manager.cc:1367] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:11.195266 8575 raft_consensus.cc:348] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:11.195361 8575 raft_consensus.cc:374] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.195407 8575 raft_consensus.cc:729] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.195540 8575 consensus_queue.cc:260] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:11.195842 8575 ts_tablet_manager.cc:1398] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:11.219323 8623 raft_consensus.cc:482] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:11.219466 8623 raft_consensus.cc:504] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.219882 8623 leader_election.cc:288] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707), 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:11.220374 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ccdd855b584d65860fd7a5777920a3" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" is_pre_election: true
I20240501 08:00:11.220511 7447 raft_consensus.cc:2454] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 0.
I20240501 08:00:11.220510 7578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ccdd855b584d65860fd7a5777920a3" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd" is_pre_election: true
I20240501 08:00:11.220700 7578 raft_consensus.cc:2454] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 0.
I20240501 08:00:11.220855 7643 leader_election.cc:302] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:11.221066 8623 raft_consensus.cc:2781] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:11.221160 8623 raft_consensus.cc:482] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:11.221204 8623 raft_consensus.cc:3037] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:11.222118 8623 raft_consensus.cc:504] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.222334 8325 raft_consensus.cc:482] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:11.222478 8623 leader_election.cc:288] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 election: Requested vote from peers d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707), 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:11.222461 8325 raft_consensus.cc:504] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:11.222803 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ccdd855b584d65860fd7a5777920a3" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
I20240501 08:00:11.222860 8325 leader_election.cc:288] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:11.222859 7578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ccdd855b584d65860fd7a5777920a3" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
I20240501 08:00:11.222914 7447 raft_consensus.cc:3037] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:11.222980 7578 raft_consensus.cc:3037] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:11.223140 7579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd17760cd5c2454db97610b2f71c0ee2" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd" is_pre_election: true
I20240501 08:00:11.223248 7579 raft_consensus.cc:2454] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 0.
I20240501 08:00:11.223311 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd17760cd5c2454db97610b2f71c0ee2" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604" is_pre_election: true
I20240501 08:00:11.223414 7708 raft_consensus.cc:2454] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 0.
I20240501 08:00:11.223533 7381 leader_election.cc:302] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:11.223695 8325 raft_consensus.cc:2781] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:11.223747 8325 raft_consensus.cc:482] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:11.223783 8325 raft_consensus.cc:3037] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:11.223937 7447 raft_consensus.cc:2454] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 1.
I20240501 08:00:11.224026 7578 raft_consensus.cc:2454] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 1.
I20240501 08:00:11.224233 7643 leader_election.cc:302] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:11.224431 8623 raft_consensus.cc:2781] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:11.224550 8623 raft_consensus.cc:686] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Becoming Leader. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Running, Role: LEADER
I20240501 08:00:11.224714 8623 consensus_queue.cc:237] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.224748 8325 raft_consensus.cc:504] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:11.225071 8325 leader_election.cc:288] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 election: Requested vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:11.225494 7578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd17760cd5c2454db97610b2f71c0ee2" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
I20240501 08:00:11.225505 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd17760cd5c2454db97610b2f71c0ee2" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604"
I20240501 08:00:11.225592 7708 raft_consensus.cc:3037] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:11.225603 7578 raft_consensus.cc:3037] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:11.226512 7578 raft_consensus.cc:2454] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 1.
I20240501 08:00:11.226516 7708 raft_consensus.cc:2454] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 1.
I20240501 08:00:11.226795 7384 leader_election.cc:302] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:11.227308 7166 catalog_manager.cc:5484] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09a7b6e2731a474b80411efb62323604 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:11.227862 8325 raft_consensus.cc:2781] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:11.227954 8325 raft_consensus.cc:686] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Becoming Leader. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Running, Role: LEADER
I20240501 08:00:11.228085 8325 consensus_queue.cc:237] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } }
I20240501 08:00:11.229934 7173 catalog_manager.cc:5484] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: term changed from 0 to 1, leader changed from <none> to d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:11.468401 8309 raft_consensus.cc:482] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:11.468571 8309 raft_consensus.cc:504] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.468994 8309 leader_election.cc:288] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707), 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:11.469586 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e178f7c3c17c4e1ca3d16b6441f76e77" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" is_pre_election: true
I20240501 08:00:11.469609 7578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e178f7c3c17c4e1ca3d16b6441f76e77" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd" is_pre_election: true
I20240501 08:00:11.469736 7447 raft_consensus.cc:2454] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 0.
I20240501 08:00:11.469758 7578 raft_consensus.cc:2454] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 0.
I20240501 08:00:11.470079 7250 leader_election.cc:302] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4f8020266b5455b872a2973fc396578, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:11.470263 8309 raft_consensus.cc:2781] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:11.470324 8309 raft_consensus.cc:482] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:11.470379 8309 raft_consensus.cc:3037] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:11.471212 8309 raft_consensus.cc:504] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.471549 8309 leader_election.cc:288] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 election: Requested vote from peers d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707), 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:11.471926 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e178f7c3c17c4e1ca3d16b6441f76e77" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
I20240501 08:00:11.471992 7578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e178f7c3c17c4e1ca3d16b6441f76e77" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
I20240501 08:00:11.472043 7447 raft_consensus.cc:3037] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:11.472155 7578 raft_consensus.cc:3037] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:11.472846 7447 raft_consensus.cc:2454] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 1.
I20240501 08:00:11.472929 7578 raft_consensus.cc:2454] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 1.
I20240501 08:00:11.473153 7250 leader_election.cc:302] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4f8020266b5455b872a2973fc396578, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:11.473315 8309 raft_consensus.cc:2781] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:11.473412 8309 raft_consensus.cc:686] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Becoming Leader. State: Replica: b4f8020266b5455b872a2973fc396578, State: Running, Role: LEADER
I20240501 08:00:11.473557 8309 consensus_queue.cc:237] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.475327 7173 catalog_manager.cc:5484] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4f8020266b5455b872a2973fc396578 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:11.653137 8620 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:11.658113 8620 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:00:11.666960 8636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:11.667224 8636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:11.670631 8636 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 08:00:11.741611 8623 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:11.749809 8678 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:11.756223 7577 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:11.756654 8309 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:11.758103 7447 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:11.758474 8309 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:11.787050 8638 mvcc.cc:205] Tried to move back new op lower bound from 7022798486548996096 to 7022798485396815872. Current Snapshot: MvccSnapshot[applied={T|T < 7022798486548996096}]
INSERT report
rows total: 2000
time total: 107.058 ms
time per row: 0.0535292 ms
I20240501 08:00:11.823416 7173 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34686:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:00:11.828929 7296 tablet_service.cc:1467] Processing CreateTablet for tablet 02b42b5fd3d440ea950b35ed461b8e12 (DEFAULT_TABLE table=rebalance_test_table_6 [id=22d40ca8e4714c2495974e28872c5687]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.829241 7296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b42b5fd3d440ea950b35ed461b8e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.829830 7690 tablet_service.cc:1467] Processing CreateTablet for tablet 02b42b5fd3d440ea950b35ed461b8e12 (DEFAULT_TABLE table=rebalance_test_table_6 [id=22d40ca8e4714c2495974e28872c5687]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.830163 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b42b5fd3d440ea950b35ed461b8e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.830227 7685 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea0db42a75a4879b3d3d608666dc2bf (DEFAULT_TABLE table=rebalance_test_table_6 [id=22d40ca8e4714c2495974e28872c5687]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.830444 7685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea0db42a75a4879b3d3d608666dc2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.829830 7688 tablet_service.cc:1467] Processing CreateTablet for tablet 8311d0921e9e417090e441884615525a (DEFAULT_TABLE table=rebalance_test_table_6 [id=22d40ca8e4714c2495974e28872c5687]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.832042 7558 tablet_service.cc:1467] Processing CreateTablet for tablet 8311d0921e9e417090e441884615525a (DEFAULT_TABLE table=rebalance_test_table_6 [id=22d40ca8e4714c2495974e28872c5687]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.832194 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8311d0921e9e417090e441884615525a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.832299 7558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8311d0921e9e417090e441884615525a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.832820 7426 tablet_service.cc:1467] Processing CreateTablet for tablet 02b42b5fd3d440ea950b35ed461b8e12 (DEFAULT_TABLE table=rebalance_test_table_6 [id=22d40ca8e4714c2495974e28872c5687]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.833034 7296 tablet_service.cc:1467] Processing CreateTablet for tablet 8311d0921e9e417090e441884615525a (DEFAULT_TABLE table=rebalance_test_table_6 [id=22d40ca8e4714c2495974e28872c5687]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.833113 7426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b42b5fd3d440ea950b35ed461b8e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.833284 7296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8311d0921e9e417090e441884615525a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.834851 8699 tablet_bootstrap.cc:492] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:11.834894 7295 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea0db42a75a4879b3d3d608666dc2bf (DEFAULT_TABLE table=rebalance_test_table_6 [id=22d40ca8e4714c2495974e28872c5687]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.835142 7295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea0db42a75a4879b3d3d608666dc2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.835561 7428 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea0db42a75a4879b3d3d608666dc2bf (DEFAULT_TABLE table=rebalance_test_table_6 [id=22d40ca8e4714c2495974e28872c5687]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:11.835794 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea0db42a75a4879b3d3d608666dc2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.835872 8699 tablet_bootstrap.cc:654] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.836983 8699 tablet_bootstrap.cc:492] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:11.837059 8699 ts_tablet_manager.cc:1367] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:11.837505 8700 tablet_bootstrap.cc:492] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:11.837534 8699 raft_consensus.cc:348] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.837641 8699 raft_consensus.cc:374] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.837679 8699 raft_consensus.cc:729] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.837879 8698 tablet_bootstrap.cc:492] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:11.837863 8699 consensus_queue.cc:260] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.838299 8699 ts_tablet_manager.cc:1398] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:11.838864 8698 tablet_bootstrap.cc:654] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.839876 8697 tablet_bootstrap.cc:492] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:11.840133 8698 tablet_bootstrap.cc:492] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:11.840265 8698 ts_tablet_manager.cc:1367] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:11.840373 8700 tablet_bootstrap.cc:654] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.840750 8697 tablet_bootstrap.cc:654] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.840957 8698 raft_consensus.cc:348] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:11.841135 8698 raft_consensus.cc:374] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.841226 8698 raft_consensus.cc:729] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.841461 8698 consensus_queue.cc:260] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:11.841593 8700 tablet_bootstrap.cc:492] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:11.841689 8700 ts_tablet_manager.cc:1367] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:00:11.841753 8697 tablet_bootstrap.cc:492] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:11.841838 8697 ts_tablet_manager.cc:1367] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:11.841894 8698 ts_tablet_manager.cc:1398] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:11.842099 8698 tablet_bootstrap.cc:492] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:11.842226 8700 raft_consensus.cc:348] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:11.842344 8700 raft_consensus.cc:374] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.842286 8697 raft_consensus.cc:348] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:11.842377 8700 raft_consensus.cc:729] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.842383 8697 raft_consensus.cc:374] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.842415 8697 raft_consensus.cc:729] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.842489 8700 consensus_queue.cc:260] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:11.842543 8697 consensus_queue.cc:260] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:11.842767 8700 ts_tablet_manager.cc:1398] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:11.842839 8697 ts_tablet_manager.cc:1398] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:11.842947 8700 tablet_bootstrap.cc:492] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:11.843003 8697 tablet_bootstrap.cc:492] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:11.843780 8697 tablet_bootstrap.cc:654] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.843828 8700 tablet_bootstrap.cc:654] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.844938 8697 tablet_bootstrap.cc:492] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:11.845013 8697 ts_tablet_manager.cc:1367] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:11.845470 8697 raft_consensus.cc:348] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.845566 8697 raft_consensus.cc:374] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.845598 8697 raft_consensus.cc:729] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.845697 8697 consensus_queue.cc:260] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.845948 8697 ts_tablet_manager.cc:1398] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:11.846159 8697 tablet_bootstrap.cc:492] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:11.846676 8698 tablet_bootstrap.cc:654] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.847004 8697 tablet_bootstrap.cc:654] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.848114 8697 tablet_bootstrap.cc:492] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:11.848170 8700 tablet_bootstrap.cc:492] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:11.848196 8697 ts_tablet_manager.cc:1367] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:11.848253 8700 ts_tablet_manager.cc:1367] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 08:00:11.848567 8698 tablet_bootstrap.cc:492] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:11.848651 8698 ts_tablet_manager.cc:1367] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 08:00:11.848610 8697 raft_consensus.cc:348] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:11.848706 8697 raft_consensus.cc:374] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.848737 8697 raft_consensus.cc:729] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.848742 8700 raft_consensus.cc:348] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:11.848891 8700 raft_consensus.cc:374] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.848852 8697 consensus_queue.cc:260] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:11.848933 8700 raft_consensus.cc:729] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.849117 8697 ts_tablet_manager.cc:1398] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:11.849071 8700 consensus_queue.cc:260] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:11.849129 8698 raft_consensus.cc:348] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.849231 8698 raft_consensus.cc:374] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.849260 8698 raft_consensus.cc:729] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.849367 8700 ts_tablet_manager.cc:1398] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:11.849412 8698 consensus_queue.cc:260] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.849682 8698 ts_tablet_manager.cc:1398] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:11.850044 8698 tablet_bootstrap.cc:492] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:11.851469 8698 tablet_bootstrap.cc:654] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:11.852576 8698 tablet_bootstrap.cc:492] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:11.852656 8698 ts_tablet_manager.cc:1367] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:00:11.852954 8698 raft_consensus.cc:348] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:11.853050 8698 raft_consensus.cc:374] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:11.853096 8698 raft_consensus.cc:729] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.853190 8698 consensus_queue.cc:260] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:11.853422 8698 ts_tablet_manager.cc:1398] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:11.921428 8614 raft_consensus.cc:482] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:11.921561 8614 raft_consensus.cc:504] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.922031 8614 leader_election.cc:288] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263), 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:11.922533 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8311d0921e9e417090e441884615525a" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578" is_pre_election: true
I20240501 08:00:11.922637 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8311d0921e9e417090e441884615525a" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604" is_pre_election: true
I20240501 08:00:11.922698 7316 raft_consensus.cc:2454] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 0.
I20240501 08:00:11.922803 7708 raft_consensus.cc:2454] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 0.
I20240501 08:00:11.923139 7515 leader_election.cc:302] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, 31b02f06be8a4455b94b1254d77005bd; no voters:
I20240501 08:00:11.923372 8614 raft_consensus.cc:2781] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:11.923439 8614 raft_consensus.cc:482] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:11.923491 8614 raft_consensus.cc:3037] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:11.924376 8614 raft_consensus.cc:504] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.924681 8614 leader_election.cc:288] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 election: Requested vote from peers b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263), 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:11.925065 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8311d0921e9e417090e441884615525a" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578"
I20240501 08:00:11.925105 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8311d0921e9e417090e441884615525a" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604"
I20240501 08:00:11.925196 7708 raft_consensus.cc:3037] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:11.925202 7316 raft_consensus.cc:3037] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:11.925951 7708 raft_consensus.cc:2454] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 1.
I20240501 08:00:11.926071 7316 raft_consensus.cc:2454] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 1.
I20240501 08:00:11.926280 7515 leader_election.cc:302] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, 31b02f06be8a4455b94b1254d77005bd; no voters:
I20240501 08:00:11.926452 8614 raft_consensus.cc:2781] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:11.926628 8614 raft_consensus.cc:686] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Becoming Leader. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Running, Role: LEADER
I20240501 08:00:11.926800 8614 consensus_queue.cc:237] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:11.927484 8309 raft_consensus.cc:482] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:11.927572 8309 raft_consensus.cc:504] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:11.927946 8309 leader_election.cc:288] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:11.928323 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b42b5fd3d440ea950b35ed461b8e12" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" is_pre_election: true
I20240501 08:00:11.928467 7447 raft_consensus.cc:2454] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 0.
I20240501 08:00:11.928761 7173 catalog_manager.cc:5484] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 31b02f06be8a4455b94b1254d77005bd (127.0.99.131). New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } }
I20240501 08:00:11.929068 7250 leader_election.cc:302] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4f8020266b5455b872a2973fc396578, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:11.929231 8309 raft_consensus.cc:2781] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:11.929296 8309 raft_consensus.cc:482] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:11.929348 8309 raft_consensus.cc:3037] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:11.929461 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b42b5fd3d440ea950b35ed461b8e12" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604" is_pre_election: true
I20240501 08:00:11.929584 7708 raft_consensus.cc:2454] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 0.
I20240501 08:00:11.930326 8309 raft_consensus.cc:504] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:11.930696 8309 leader_election.cc:288] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 election: Requested vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:11.930950 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b42b5fd3d440ea950b35ed461b8e12" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604"
I20240501 08:00:11.931041 7708 raft_consensus.cc:3037] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:11.931286 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b42b5fd3d440ea950b35ed461b8e12" candidate_uuid: "b4f8020266b5455b872a2973fc396578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
I20240501 08:00:11.931403 7447 raft_consensus.cc:3037] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:11.932053 7708 raft_consensus.cc:2454] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 1.
I20240501 08:00:11.932351 7447 raft_consensus.cc:2454] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f8020266b5455b872a2973fc396578 in term 1.
I20240501 08:00:11.932417 7253 leader_election.cc:302] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:11.932595 8543 raft_consensus.cc:2781] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:11.932677 8543 raft_consensus.cc:686] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Becoming Leader. State: Replica: b4f8020266b5455b872a2973fc396578, State: Running, Role: LEADER
I20240501 08:00:11.932814 8543 consensus_queue.cc:237] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:11.934521 7173 catalog_manager.cc:5484] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4f8020266b5455b872a2973fc396578 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:12.154175 8695 raft_consensus.cc:482] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:12.154318 8695 raft_consensus.cc:504] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.154731 8695 leader_election.cc:288] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:12.155243 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea0db42a75a4879b3d3d608666dc2bf" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578" is_pre_election: true
I20240501 08:00:12.155299 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea0db42a75a4879b3d3d608666dc2bf" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" is_pre_election: true
I20240501 08:00:12.155421 7316 raft_consensus.cc:2454] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 0.
I20240501 08:00:12.155464 7447 raft_consensus.cc:2454] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 0.
I20240501 08:00:12.155812 7646 leader_election.cc:302] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:12.155987 8695 raft_consensus.cc:2781] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:12.156054 8695 raft_consensus.cc:482] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:12.156105 8695 raft_consensus.cc:3037] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:12.156949 8695 raft_consensus.cc:504] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.157258 8695 leader_election.cc:288] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 election: Requested vote from peers d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:12.157605 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea0db42a75a4879b3d3d608666dc2bf" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578"
I20240501 08:00:12.157588 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea0db42a75a4879b3d3d608666dc2bf" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
I20240501 08:00:12.157707 7447 raft_consensus.cc:3037] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:12.157718 7316 raft_consensus.cc:3037] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:12.158494 7447 raft_consensus.cc:2454] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 1.
I20240501 08:00:12.158653 7316 raft_consensus.cc:2454] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 1.
I20240501 08:00:12.158766 7643 leader_election.cc:302] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:12.158948 8695 raft_consensus.cc:2781] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:12.159027 8695 raft_consensus.cc:686] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Becoming Leader. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Running, Role: LEADER
I20240501 08:00:12.159169 8695 consensus_queue.cc:237] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.165823 7173 catalog_manager.cc:5484] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09a7b6e2731a474b80411efb62323604 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: UNKNOWN } } }
W20240501 08:00:12.311878 8714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:12.312152 8714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:12.315496 8714 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 08:00:12.417244 7447 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:12.417728 8543 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:12.418177 7316 raft_consensus.cc:1261] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:12.418277 7447 raft_consensus.cc:1261] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:12.418605 8695 consensus_queue.cc:1035] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:12.418874 8695 consensus_queue.cc:1035] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:12.423318 7708 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:12.423724 8309 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:12.437137 7708 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:12.438453 8614 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:12.441004 7317 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:12.442703 8614 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:12.447362 8764 mvcc.cc:205] Tried to move back new op lower bound from 7022798489255673856 to 7022798487277998080. Current Snapshot: MvccSnapshot[applied={T|T < 7022798489255673856}]
I20240501 08:00:12.462597 8748 mvcc.cc:205] Tried to move back new op lower bound from 7022798489323556864 to 7022798487253311488. Current Snapshot: MvccSnapshot[applied={T|T < 7022798489323556864}]
INSERT report
rows total: 2000
time total: 125.323 ms
time per row: 0.0626617 ms
I20240501 08:00:12.498282 7171 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34686:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:00:12.504750 7295 tablet_service.cc:1467] Processing CreateTablet for tablet 8882528b26974c7aa73bb0ff0c444836 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c7ddae9ad2184cb9bd097d68171c3fc4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.505110 7295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8882528b26974c7aa73bb0ff0c444836. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.505117 7685 tablet_service.cc:1467] Processing CreateTablet for tablet f647cfc562794e65a05918b241014d72 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c7ddae9ad2184cb9bd097d68171c3fc4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.505313 7428 tablet_service.cc:1467] Processing CreateTablet for tablet 8882528b26974c7aa73bb0ff0c444836 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c7ddae9ad2184cb9bd097d68171c3fc4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.505313 7688 tablet_service.cc:1467] Processing CreateTablet for tablet df6dd3d8482d4931b1d3014a67e739c6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c7ddae9ad2184cb9bd097d68171c3fc4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.505390 7685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f647cfc562794e65a05918b241014d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.505576 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6dd3d8482d4931b1d3014a67e739c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.505578 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8882528b26974c7aa73bb0ff0c444836. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.507896 8777 tablet_bootstrap.cc:492] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:12.508193 8776 tablet_bootstrap.cc:492] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:12.508211 7558 tablet_service.cc:1467] Processing CreateTablet for tablet 8882528b26974c7aa73bb0ff0c444836 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c7ddae9ad2184cb9bd097d68171c3fc4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.508483 7558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8882528b26974c7aa73bb0ff0c444836. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.508781 8777 tablet_bootstrap.cc:654] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.509181 8776 tablet_bootstrap.cc:654] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.509784 8777 tablet_bootstrap.cc:492] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:12.509862 8775 tablet_bootstrap.cc:492] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:12.509969 8777 ts_tablet_manager.cc:1367] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:12.510442 8777 raft_consensus.cc:348] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:12.510535 8777 raft_consensus.cc:374] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.510571 8777 raft_consensus.cc:729] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.508211 7559 tablet_service.cc:1467] Processing CreateTablet for tablet f647cfc562794e65a05918b241014d72 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c7ddae9ad2184cb9bd097d68171c3fc4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.510722 8777 consensus_queue.cc:260] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:12.510880 7559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f647cfc562794e65a05918b241014d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.511008 8777 ts_tablet_manager.cc:1398] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:12.511482 8776 tablet_bootstrap.cc:492] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:12.511567 8776 ts_tablet_manager.cc:1367] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:00:12.512179 8776 raft_consensus.cc:348] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:12.512287 8776 raft_consensus.cc:374] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.512317 8776 raft_consensus.cc:729] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.512432 8775 tablet_bootstrap.cc:654] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.513101 8776 consensus_queue.cc:260] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:12.513756 8775 tablet_bootstrap.cc:492] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:12.513883 8775 ts_tablet_manager.cc:1367] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:00:12.514343 7559 tablet_service.cc:1467] Processing CreateTablet for tablet df6dd3d8482d4931b1d3014a67e739c6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c7ddae9ad2184cb9bd097d68171c3fc4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.514379 7295 tablet_service.cc:1467] Processing CreateTablet for tablet df6dd3d8482d4931b1d3014a67e739c6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c7ddae9ad2184cb9bd097d68171c3fc4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.514384 8775 raft_consensus.cc:348] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.514494 8775 raft_consensus.cc:374] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.514532 8775 raft_consensus.cc:729] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.514572 7295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6dd3d8482d4931b1d3014a67e739c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.514570 7559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6dd3d8482d4931b1d3014a67e739c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.514654 8775 consensus_queue.cc:260] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.514926 8775 ts_tablet_manager.cc:1398] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:12.515071 8775 tablet_bootstrap.cc:492] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:12.515931 8775 tablet_bootstrap.cc:654] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.513754 7428 tablet_service.cc:1467] Processing CreateTablet for tablet f647cfc562794e65a05918b241014d72 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c7ddae9ad2184cb9bd097d68171c3fc4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.516534 8777 tablet_bootstrap.cc:492] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:12.516623 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f647cfc562794e65a05918b241014d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.516963 8780 tablet_bootstrap.cc:492] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:12.517433 8777 tablet_bootstrap.cc:654] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.517771 8780 tablet_bootstrap.cc:654] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.518774 8780 tablet_bootstrap.cc:492] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:12.518853 8780 ts_tablet_manager.cc:1367] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:00:12.518853 8777 tablet_bootstrap.cc:492] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:12.518935 8777 ts_tablet_manager.cc:1367] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:00:12.519281 8780 raft_consensus.cc:348] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:12.519384 8780 raft_consensus.cc:374] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.519416 8780 raft_consensus.cc:729] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.519359 8777 raft_consensus.cc:348] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.519462 8777 raft_consensus.cc:374] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.519519 8777 raft_consensus.cc:729] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.519531 8780 consensus_queue.cc:260] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:12.519578 8776 ts_tablet_manager.cc:1398] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 08:00:12.519728 8776 tablet_bootstrap.cc:492] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:12.519696 8777 consensus_queue.cc:260] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.519816 8780 ts_tablet_manager.cc:1398] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:12.519970 8780 tablet_bootstrap.cc:492] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:12.520197 8777 ts_tablet_manager.cc:1398] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:12.520776 8780 tablet_bootstrap.cc:654] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.521531 8776 tablet_bootstrap.cc:654] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.522469 8775 tablet_bootstrap.cc:492] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:12.522614 8775 ts_tablet_manager.cc:1367] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 08:00:12.522701 8780 tablet_bootstrap.cc:492] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:12.522776 8780 ts_tablet_manager.cc:1367] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 08:00:12.523144 8780 raft_consensus.cc:348] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:12.523252 8780 raft_consensus.cc:374] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.523156 8775 raft_consensus.cc:348] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:12.523295 8780 raft_consensus.cc:729] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.523330 8775 raft_consensus.cc:374] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.523391 8775 raft_consensus.cc:729] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.523563 8775 consensus_queue.cc:260] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:12.524044 8775 ts_tablet_manager.cc:1398] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:12.524021 8780 consensus_queue.cc:260] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:12.524384 8780 ts_tablet_manager.cc:1398] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:00:12.524528 8780 tablet_bootstrap.cc:492] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:12.525390 8780 tablet_bootstrap.cc:654] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.526863 8780 tablet_bootstrap.cc:492] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:12.526954 8780 ts_tablet_manager.cc:1367] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:00:12.527387 8780 raft_consensus.cc:348] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.527501 8780 raft_consensus.cc:374] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.527542 8780 raft_consensus.cc:729] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.527613 8776 tablet_bootstrap.cc:492] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:12.527694 8776 ts_tablet_manager.cc:1367] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240501 08:00:12.527664 8780 consensus_queue.cc:260] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.527943 8780 ts_tablet_manager.cc:1398] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:12.528113 8776 raft_consensus.cc:348] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:12.528210 8776 raft_consensus.cc:374] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.528232 8776 raft_consensus.cc:729] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.528326 8776 consensus_queue.cc:260] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:12.528641 8776 ts_tablet_manager.cc:1398] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:12.529973 8678 raft_consensus.cc:482] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:12.530115 8678 raft_consensus.cc:504] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.530548 8678 leader_election.cc:288] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:12.530854 7577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6dd3d8482d4931b1d3014a67e739c6" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd" is_pre_election: true
I20240501 08:00:12.530876 7317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6dd3d8482d4931b1d3014a67e739c6" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578" is_pre_election: true
I20240501 08:00:12.530958 7577 raft_consensus.cc:2454] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 0.
I20240501 08:00:12.531020 7317 raft_consensus.cc:2454] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 0.
I20240501 08:00:12.531235 7643 leader_election.cc:302] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, 31b02f06be8a4455b94b1254d77005bd; no voters:
I20240501 08:00:12.531406 8678 raft_consensus.cc:2781] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:12.531458 8678 raft_consensus.cc:482] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:12.531483 8678 raft_consensus.cc:3037] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:12.532335 8678 raft_consensus.cc:504] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.532670 8678 leader_election.cc:288] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 election: Requested vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:12.533007 7577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6dd3d8482d4931b1d3014a67e739c6" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
I20240501 08:00:12.533108 7577 raft_consensus.cc:3037] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:12.533134 7317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6dd3d8482d4931b1d3014a67e739c6" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578"
I20240501 08:00:12.533229 7317 raft_consensus.cc:3037] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:12.534104 7577 raft_consensus.cc:2454] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 1.
I20240501 08:00:12.534219 7317 raft_consensus.cc:2454] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 1.
I20240501 08:00:12.534410 7643 leader_election.cc:302] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, 31b02f06be8a4455b94b1254d77005bd; no voters:
I20240501 08:00:12.534560 8695 raft_consensus.cc:2781] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:12.534662 8695 raft_consensus.cc:686] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Becoming Leader. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Running, Role: LEADER
I20240501 08:00:12.534795 8695 consensus_queue.cc:237] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.536783 7171 catalog_manager.cc:5484] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09a7b6e2731a474b80411efb62323604 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:12.620551 8620 raft_consensus.cc:482] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:12.620690 8620 raft_consensus.cc:504] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:12.621086 8620 leader_election.cc:288] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263), 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:12.621646 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8882528b26974c7aa73bb0ff0c444836" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578" is_pre_election: true
I20240501 08:00:12.621646 7577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8882528b26974c7aa73bb0ff0c444836" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd" is_pre_election: true
I20240501 08:00:12.621803 7577 raft_consensus.cc:2454] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 0.
I20240501 08:00:12.621803 7316 raft_consensus.cc:2454] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 0.
I20240501 08:00:12.622131 7381 leader_election.cc:302] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:12.622335 8620 raft_consensus.cc:2781] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:12.622414 8620 raft_consensus.cc:482] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:12.622469 8620 raft_consensus.cc:3037] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:12.623356 8620 raft_consensus.cc:504] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:12.623687 8620 leader_election.cc:288] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 election: Requested vote from peers b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263), 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:12.624042 7577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8882528b26974c7aa73bb0ff0c444836" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
I20240501 08:00:12.624064 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8882528b26974c7aa73bb0ff0c444836" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578"
I20240501 08:00:12.624172 7316 raft_consensus.cc:3037] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:12.624182 7577 raft_consensus.cc:3037] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:12.624898 7577 raft_consensus.cc:2454] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 1.
I20240501 08:00:12.624914 7316 raft_consensus.cc:2454] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 1.
I20240501 08:00:12.625191 7384 leader_election.cc:302] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4f8020266b5455b872a2973fc396578, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:12.625384 8620 raft_consensus.cc:2781] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:12.625586 8620 raft_consensus.cc:686] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Becoming Leader. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Running, Role: LEADER
I20240501 08:00:12.625722 8620 consensus_queue.cc:237] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:12.627525 7171 catalog_manager.cc:5484] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: term changed from 0 to 1, leader changed from <none> to d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:12.651985 8620 raft_consensus.cc:482] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:12.652133 8620 raft_consensus.cc:504] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:12.652568 8620 leader_election.cc:288] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:12.652952 7577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f647cfc562794e65a05918b241014d72" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd" is_pre_election: true
I20240501 08:00:12.653004 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f647cfc562794e65a05918b241014d72" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604" is_pre_election: true
I20240501 08:00:12.653107 7577 raft_consensus.cc:2454] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 0.
I20240501 08:00:12.653124 7708 raft_consensus.cc:2454] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 0.
I20240501 08:00:12.653407 7381 leader_election.cc:302] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:12.653631 8620 raft_consensus.cc:2781] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:12.653698 8620 raft_consensus.cc:482] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:12.653738 8620 raft_consensus.cc:3037] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:12.654665 8620 raft_consensus.cc:504] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:12.654951 8620 leader_election.cc:288] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 election: Requested vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:12.655301 7577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f647cfc562794e65a05918b241014d72" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
I20240501 08:00:12.655303 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f647cfc562794e65a05918b241014d72" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604"
I20240501 08:00:12.655423 7577 raft_consensus.cc:3037] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:12.655431 7708 raft_consensus.cc:3037] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:12.656343 7708 raft_consensus.cc:2454] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 1.
I20240501 08:00:12.656348 7577 raft_consensus.cc:2454] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 1.
I20240501 08:00:12.656626 7381 leader_election.cc:302] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:12.656769 8620 raft_consensus.cc:2781] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:12.656857 8620 raft_consensus.cc:686] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Becoming Leader. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Running, Role: LEADER
I20240501 08:00:12.656996 8620 consensus_queue.cc:237] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:12.658847 7171 catalog_manager.cc:5484] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: term changed from 0 to 1, leader changed from <none> to d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } }
W20240501 08:00:12.776182 8792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:12.776458 8792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:12.779711 8792 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 08:00:12.871407 7577 raft_consensus.cc:1261] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:12.871668 7316 raft_consensus.cc:1261] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:12.872002 7577 raft_consensus.cc:1261] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:12.872421 7708 raft_consensus.cc:1261] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:12.872416 8620 consensus_queue.cc:1035] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:12.872709 8620 consensus_queue.cc:1035] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:12.872932 8620 consensus_queue.cc:1035] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:12.873168 8620 consensus_queue.cc:1035] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:12.875924 7317 raft_consensus.cc:1261] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:12.876394 8695 consensus_queue.cc:1035] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:12.877691 7577 raft_consensus.cc:1261] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:12.882786 8678 consensus_queue.cc:1035] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:12.907600 8745 mvcc.cc:205] Tried to move back new op lower bound from 7022798491117133824 to 7022798490116386816. Current Snapshot: MvccSnapshot[applied={T|T < 7022798491117133824 or (T in {7022798491117133824})}]
INSERT report
rows total: 2000
time total: 112.805 ms
time per row: 0.0564023 ms
I20240501 08:00:12.940286 7173 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34686:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:00:12.945881 7428 tablet_service.cc:1467] Processing CreateTablet for tablet cd07a00b6817400f86bf8c32bd592d8b (DEFAULT_TABLE table=rebalance_test_table_8 [id=82bf711832394b1abda2824bb95c9358]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.945905 7559 tablet_service.cc:1467] Processing CreateTablet for tablet cd07a00b6817400f86bf8c32bd592d8b (DEFAULT_TABLE table=rebalance_test_table_8 [id=82bf711832394b1abda2824bb95c9358]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.946282 7559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd07a00b6817400f86bf8c32bd592d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.946287 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd07a00b6817400f86bf8c32bd592d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.946545 7688 tablet_service.cc:1467] Processing CreateTablet for tablet ffc23756b44d4d588e0a8175075d9bc6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=82bf711832394b1abda2824bb95c9358]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.946791 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc23756b44d4d588e0a8175075d9bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.946784 7558 tablet_service.cc:1467] Processing CreateTablet for tablet 5d18a2918c75467a9fe59979ce8936cd (DEFAULT_TABLE table=rebalance_test_table_8 [id=82bf711832394b1abda2824bb95c9358]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.946979 7558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d18a2918c75467a9fe59979ce8936cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.948495 8776 tablet_bootstrap.cc:492] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:12.948508 8780 tablet_bootstrap.cc:492] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:12.948987 7685 tablet_service.cc:1467] Processing CreateTablet for tablet 5d18a2918c75467a9fe59979ce8936cd (DEFAULT_TABLE table=rebalance_test_table_8 [id=82bf711832394b1abda2824bb95c9358]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.949204 7685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d18a2918c75467a9fe59979ce8936cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.949273 8780 tablet_bootstrap.cc:654] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.949342 8776 tablet_bootstrap.cc:654] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.950493 8780 tablet_bootstrap.cc:492] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:12.950502 7295 tablet_service.cc:1467] Processing CreateTablet for tablet cd07a00b6817400f86bf8c32bd592d8b (DEFAULT_TABLE table=rebalance_test_table_8 [id=82bf711832394b1abda2824bb95c9358]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.950580 8780 ts_tablet_manager.cc:1367] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:00:12.950755 7295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd07a00b6817400f86bf8c32bd592d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.950775 7426 tablet_service.cc:1467] Processing CreateTablet for tablet 5d18a2918c75467a9fe59979ce8936cd (DEFAULT_TABLE table=rebalance_test_table_8 [id=82bf711832394b1abda2824bb95c9358]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.950994 7426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d18a2918c75467a9fe59979ce8936cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.951013 8780 raft_consensus.cc:348] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.951110 8780 raft_consensus.cc:374] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.951139 8780 raft_consensus.cc:729] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.951193 8775 tablet_bootstrap.cc:492] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:12.951248 8780 consensus_queue.cc:260] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.951498 8780 ts_tablet_manager.cc:1398] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:12.951670 8780 tablet_bootstrap.cc:492] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:12.952069 8775 tablet_bootstrap.cc:654] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.952508 8780 tablet_bootstrap.cc:654] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.953011 8777 tablet_bootstrap.cc:492] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:12.953079 7296 tablet_service.cc:1467] Processing CreateTablet for tablet ffc23756b44d4d588e0a8175075d9bc6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=82bf711832394b1abda2824bb95c9358]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.953305 7296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc23756b44d4d588e0a8175075d9bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.953658 7428 tablet_service.cc:1467] Processing CreateTablet for tablet ffc23756b44d4d588e0a8175075d9bc6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=82bf711832394b1abda2824bb95c9358]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:12.953868 8777 tablet_bootstrap.cc:654] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.953883 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc23756b44d4d588e0a8175075d9bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:12.954563 8775 tablet_bootstrap.cc:492] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:12.954638 8775 ts_tablet_manager.cc:1367] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:00:12.955075 8775 raft_consensus.cc:348] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:12.955188 8775 raft_consensus.cc:374] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.955240 8775 raft_consensus.cc:729] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.955380 8775 consensus_queue.cc:260] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:12.955689 8775 ts_tablet_manager.cc:1398] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:12.955830 8775 tablet_bootstrap.cc:492] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:12.956518 8780 tablet_bootstrap.cc:492] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:12.956619 8780 ts_tablet_manager.cc:1367] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 08:00:12.957144 8775 tablet_bootstrap.cc:654] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.957095 8780 raft_consensus.cc:348] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:12.957219 8780 raft_consensus.cc:374] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.957258 8780 raft_consensus.cc:729] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.957382 8780 consensus_queue.cc:260] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:12.957544 8776 tablet_bootstrap.cc:492] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:12.957618 8776 ts_tablet_manager.cc:1367] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20240501 08:00:12.957676 8780 ts_tablet_manager.cc:1398] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:12.958202 8777 tablet_bootstrap.cc:492] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:12.958283 8777 ts_tablet_manager.cc:1367] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 08:00:12.958351 8776 raft_consensus.cc:348] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.958470 8776 raft_consensus.cc:374] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.958511 8776 raft_consensus.cc:729] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.958628 8776 consensus_queue.cc:260] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.958890 8776 ts_tablet_manager.cc:1398] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 08:00:12.958858 8777 raft_consensus.cc:348] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.958953 8777 raft_consensus.cc:374] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.958985 8777 raft_consensus.cc:729] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.959056 8776 tablet_bootstrap.cc:492] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:12.959102 8777 consensus_queue.cc:260] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:12.959354 8777 ts_tablet_manager.cc:1398] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:12.959522 8777 tablet_bootstrap.cc:492] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:12.959826 8776 tablet_bootstrap.cc:654] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.960459 8777 tablet_bootstrap.cc:654] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.960703 8775 tablet_bootstrap.cc:492] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:12.960793 8775 ts_tablet_manager.cc:1367] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:00:12.961148 8775 raft_consensus.cc:348] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:12.961257 8775 raft_consensus.cc:374] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.961300 8775 raft_consensus.cc:729] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.961439 8775 consensus_queue.cc:260] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:12.961544 8777 tablet_bootstrap.cc:492] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:12.961628 8777 ts_tablet_manager.cc:1367] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:12.961730 8775 ts_tablet_manager.cc:1398] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:12.962059 8777 raft_consensus.cc:348] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:12.962167 8777 raft_consensus.cc:374] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.962203 8777 raft_consensus.cc:729] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.962335 8777 consensus_queue.cc:260] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:12.962623 8777 ts_tablet_manager.cc:1398] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:12.964821 8776 tablet_bootstrap.cc:492] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:12.964901 8776 ts_tablet_manager.cc:1367] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 08:00:12.965334 8776 raft_consensus.cc:348] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:12.965431 8776 raft_consensus.cc:374] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.965453 8776 raft_consensus.cc:729] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.965553 8776 consensus_queue.cc:260] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:12.965818 8776 ts_tablet_manager.cc:1398] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:12.966045 8776 tablet_bootstrap.cc:492] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:12.966955 8776 tablet_bootstrap.cc:654] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:12.967970 8776 tablet_bootstrap.cc:492] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:12.968055 8776 ts_tablet_manager.cc:1367] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:12.968453 8776 raft_consensus.cc:348] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:12.968571 8776 raft_consensus.cc:374] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:12.968616 8776 raft_consensus.cc:729] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.968730 8776 consensus_queue.cc:260] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:12.968997 8776 ts_tablet_manager.cc:1398] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:13.007905 8819 raft_consensus.cc:482] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:13.008064 8819 raft_consensus.cc:504] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:13.008525 8819 leader_election.cc:288] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:13.009006 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc23756b44d4d588e0a8175075d9bc6" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604" is_pre_election: true
I20240501 08:00:13.009076 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc23756b44d4d588e0a8175075d9bc6" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578" is_pre_election: true
I20240501 08:00:13.009182 7708 raft_consensus.cc:2454] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 0.
I20240501 08:00:13.009217 7316 raft_consensus.cc:2454] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 0.
I20240501 08:00:13.009486 7384 leader_election.cc:302] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:13.009776 8819 raft_consensus.cc:2781] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:13.009831 8819 raft_consensus.cc:482] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:13.009873 8819 raft_consensus.cc:3037] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.010964 8819 raft_consensus.cc:504] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:13.011349 8819 leader_election.cc:288] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 election: Requested vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:13.011706 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc23756b44d4d588e0a8175075d9bc6" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604"
I20240501 08:00:13.011824 7708 raft_consensus.cc:3037] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.011796 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc23756b44d4d588e0a8175075d9bc6" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578"
I20240501 08:00:13.011926 7316 raft_consensus.cc:3037] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.012763 7316 raft_consensus.cc:2454] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 1.
I20240501 08:00:13.012771 7708 raft_consensus.cc:2454] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 1.
I20240501 08:00:13.013075 7384 leader_election.cc:302] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:13.013239 8819 raft_consensus.cc:2781] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:13.013324 8819 raft_consensus.cc:686] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Becoming Leader. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Running, Role: LEADER
I20240501 08:00:13.013478 8819 consensus_queue.cc:237] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 08:00:13.015239 7171 catalog_manager.cc:5484] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: term changed from 0 to 1, leader changed from <none> to d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } }
I20240501 08:00:13.122229 8772 raft_consensus.cc:482] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:13.122375 8772 raft_consensus.cc:504] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.122882 8772 leader_election.cc:288] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:13.123313 7448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd07a00b6817400f86bf8c32bd592d8b" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" is_pre_election: true
I20240501 08:00:13.123337 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd07a00b6817400f86bf8c32bd592d8b" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578" is_pre_election: true
I20240501 08:00:13.123463 7316 raft_consensus.cc:2454] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 0.
I20240501 08:00:13.123471 7448 raft_consensus.cc:2454] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 0.
I20240501 08:00:13.123761 7515 leader_election.cc:302] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:13.123943 8772 raft_consensus.cc:2781] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:13.124011 8772 raft_consensus.cc:482] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:13.124045 8772 raft_consensus.cc:3037] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.124871 8772 raft_consensus.cc:504] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.125198 8772 leader_election.cc:288] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 election: Requested vote from peers d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:13.125578 7448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd07a00b6817400f86bf8c32bd592d8b" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
I20240501 08:00:13.125602 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd07a00b6817400f86bf8c32bd592d8b" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578"
I20240501 08:00:13.125687 7448 raft_consensus.cc:3037] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.125721 7316 raft_consensus.cc:3037] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.126543 7316 raft_consensus.cc:2454] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 1.
I20240501 08:00:13.126564 7448 raft_consensus.cc:2454] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 1.
I20240501 08:00:13.126863 7512 leader_election.cc:302] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:13.127032 8772 raft_consensus.cc:2781] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:13.127321 8772 raft_consensus.cc:686] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Becoming Leader. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Running, Role: LEADER
I20240501 08:00:13.127524 8772 consensus_queue.cc:237] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.129273 7171 catalog_manager.cc:5484] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 31b02f06be8a4455b94b1254d77005bd (127.0.99.131). New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:13.280143 8819 raft_consensus.cc:482] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:13.280283 8819 raft_consensus.cc:504] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:13.280746 8819 leader_election.cc:288] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:13.281199 7709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d18a2918c75467a9fe59979ce8936cd" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604" is_pre_election: true
I20240501 08:00:13.281224 7577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d18a2918c75467a9fe59979ce8936cd" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd" is_pre_election: true
I20240501 08:00:13.281373 7577 raft_consensus.cc:2454] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 0.
I20240501 08:00:13.281345 7709 raft_consensus.cc:2454] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 0.
I20240501 08:00:13.281700 7384 leader_election.cc:302] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:13.281885 8819 raft_consensus.cc:2781] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:13.281941 8819 raft_consensus.cc:482] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:13.281971 8819 raft_consensus.cc:3037] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.282878 8819 raft_consensus.cc:504] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:13.283211 8819 leader_election.cc:288] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 election: Requested vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:13.283648 7577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d18a2918c75467a9fe59979ce8936cd" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
I20240501 08:00:13.283756 7577 raft_consensus.cc:3037] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.284547 7577 raft_consensus.cc:2454] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 1.
I20240501 08:00:13.284782 7709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d18a2918c75467a9fe59979ce8936cd" candidate_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604"
I20240501 08:00:13.284829 7381 leader_election.cc:302] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:13.284893 7709 raft_consensus.cc:3037] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.285006 8819 raft_consensus.cc:2781] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:13.285079 8819 raft_consensus.cc:686] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Becoming Leader. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Running, Role: LEADER
I20240501 08:00:13.285228 8819 consensus_queue.cc:237] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:13.285904 7709 raft_consensus.cc:2454] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b4be7bbd1649d0b7c5e8efe81f759f in term 1.
I20240501 08:00:13.287210 7171 catalog_manager.cc:5484] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: term changed from 0 to 1, leader changed from <none> to d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } }
W20240501 08:00:13.445730 8843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:13.445976 8843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:13.449139 8843 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 08:00:13.525758 8819 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:13.531533 8789 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:13.541153 7316 raft_consensus.cc:1261] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:13.541206 7448 raft_consensus.cc:1261] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:13.541586 8840 consensus_queue.cc:1035] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:13.542040 8772 consensus_queue.cc:1035] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:13.545094 7577 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:13.547583 8789 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:13.549587 7709 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:13.549937 8789 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 123.747 ms
time per row: 0.0618733 ms
I20240501 08:00:13.616608 7175 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34686:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:00:13.622642 7685 tablet_service.cc:1467] Processing CreateTablet for tablet 28baea6d7d3f442ba1dbfc5b37de90ac (DEFAULT_TABLE table=rebalance_test_table_9 [id=15bea2064e1c414fb23182f6bc324bfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:13.622763 7558 tablet_service.cc:1467] Processing CreateTablet for tablet 28baea6d7d3f442ba1dbfc5b37de90ac (DEFAULT_TABLE table=rebalance_test_table_9 [id=15bea2064e1c414fb23182f6bc324bfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:13.623006 7558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28baea6d7d3f442ba1dbfc5b37de90ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:13.623001 7685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28baea6d7d3f442ba1dbfc5b37de90ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:13.623354 7296 tablet_service.cc:1467] Processing CreateTablet for tablet 28baea6d7d3f442ba1dbfc5b37de90ac (DEFAULT_TABLE table=rebalance_test_table_9 [id=15bea2064e1c414fb23182f6bc324bfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:13.623644 7296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28baea6d7d3f442ba1dbfc5b37de90ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:13.625363 8905 tablet_bootstrap.cc:492] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:13.625519 7688 tablet_service.cc:1467] Processing CreateTablet for tablet 1e44704a9b9c46cf954d8e6937983841 (DEFAULT_TABLE table=rebalance_test_table_9 [id=15bea2064e1c414fb23182f6bc324bfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:13.625622 7295 tablet_service.cc:1467] Processing CreateTablet for tablet 1e44704a9b9c46cf954d8e6937983841 (DEFAULT_TABLE table=rebalance_test_table_9 [id=15bea2064e1c414fb23182f6bc324bfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:13.625767 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e44704a9b9c46cf954d8e6937983841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:13.625839 8907 tablet_bootstrap.cc:492] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:13.626107 7297 tablet_service.cc:1467] Processing CreateTablet for tablet 4297f95243ac431099a8279c3d9e4b25 (DEFAULT_TABLE table=rebalance_test_table_9 [id=15bea2064e1c414fb23182f6bc324bfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:13.626624 8905 tablet_bootstrap.cc:654] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:13.625845 7295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e44704a9b9c46cf954d8e6937983841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:13.627027 7297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4297f95243ac431099a8279c3d9e4b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:13.628906 8906 tablet_bootstrap.cc:492] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:13.628957 7559 tablet_service.cc:1467] Processing CreateTablet for tablet 4297f95243ac431099a8279c3d9e4b25 (DEFAULT_TABLE table=rebalance_test_table_9 [id=15bea2064e1c414fb23182f6bc324bfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:13.629024 7428 tablet_service.cc:1467] Processing CreateTablet for tablet 4297f95243ac431099a8279c3d9e4b25 (DEFAULT_TABLE table=rebalance_test_table_9 [id=15bea2064e1c414fb23182f6bc324bfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:13.629199 7559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4297f95243ac431099a8279c3d9e4b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:13.629290 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4297f95243ac431099a8279c3d9e4b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:13.629772 8906 tablet_bootstrap.cc:654] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:13.629819 8907 tablet_bootstrap.cc:654] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:13.630960 8907 tablet_bootstrap.cc:492] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:13.631047 8907 ts_tablet_manager.cc:1367] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:00:13.631464 8906 tablet_bootstrap.cc:492] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:13.631546 8906 ts_tablet_manager.cc:1367] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:00:13.631543 8907 raft_consensus.cc:348] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.631659 8907 raft_consensus.cc:374] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:13.631696 8907 raft_consensus.cc:729] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:13.631834 8907 consensus_queue.cc:260] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.631979 8906 raft_consensus.cc:348] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.632100 8906 raft_consensus.cc:374] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:13.632102 8907 ts_tablet_manager.cc:1398] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:13.632124 8906 raft_consensus.cc:729] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:13.632230 8906 consensus_queue.cc:260] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.632265 8907 tablet_bootstrap.cc:492] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:13.632403 7558 tablet_service.cc:1467] Processing CreateTablet for tablet 1e44704a9b9c46cf954d8e6937983841 (DEFAULT_TABLE table=rebalance_test_table_9 [id=15bea2064e1c414fb23182f6bc324bfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:00:13.632479 8906 ts_tablet_manager.cc:1398] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:13.632617 7558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e44704a9b9c46cf954d8e6937983841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:13.632627 8906 tablet_bootstrap.cc:492] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:13.633106 8907 tablet_bootstrap.cc:654] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:13.633507 8906 tablet_bootstrap.cc:654] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:13.634290 8911 tablet_bootstrap.cc:492] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:00:13.634886 8906 tablet_bootstrap.cc:492] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604: No bootstrap required, opened a new log
I20240501 08:00:13.634967 8906 ts_tablet_manager.cc:1367] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:00:13.635404 8906 raft_consensus.cc:348] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.635500 8906 raft_consensus.cc:374] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:13.635538 8906 raft_consensus.cc:729] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: FOLLOWER
I20240501 08:00:13.635648 8907 tablet_bootstrap.cc:492] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:13.635658 8906 consensus_queue.cc:260] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.635747 8907 ts_tablet_manager.cc:1367] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:00:13.635943 8906 ts_tablet_manager.cc:1398] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:13.636178 8907 raft_consensus.cc:348] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.636281 8907 raft_consensus.cc:374] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:13.636315 8907 raft_consensus.cc:729] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:13.636431 8907 consensus_queue.cc:260] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.636693 8907 ts_tablet_manager.cc:1398] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:13.636839 8907 tablet_bootstrap.cc:492] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:00:13.637096 8911 tablet_bootstrap.cc:654] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:13.637822 8907 tablet_bootstrap.cc:654] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:13.638576 8905 tablet_bootstrap.cc:492] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:13.638720 8905 ts_tablet_manager.cc:1367] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240501 08:00:13.638911 8911 tablet_bootstrap.cc:492] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f: No bootstrap required, opened a new log
I20240501 08:00:13.638908 8907 tablet_bootstrap.cc:492] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578: No bootstrap required, opened a new log
I20240501 08:00:13.638990 8907 ts_tablet_manager.cc:1367] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:00:13.639104 8911 ts_tablet_manager.cc:1367] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:00:13.639318 8905 raft_consensus.cc:348] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.639477 8905 raft_consensus.cc:374] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:13.639545 8905 raft_consensus.cc:729] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:13.639703 8905 consensus_queue.cc:260] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.639716 8907 raft_consensus.cc:348] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:13.639811 8907 raft_consensus.cc:374] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:13.639847 8907 raft_consensus.cc:729] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: FOLLOWER
I20240501 08:00:13.639737 8911 raft_consensus.cc:348] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:13.639976 8911 raft_consensus.cc:374] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:13.640012 8911 raft_consensus.cc:729] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: FOLLOWER
I20240501 08:00:13.639964 8907 consensus_queue.cc:260] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:13.640031 8905 ts_tablet_manager.cc:1398] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:13.640261 8907 ts_tablet_manager.cc:1398] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:13.640162 8911 consensus_queue.cc:260] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:13.640398 8905 tablet_bootstrap.cc:492] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:13.640579 8911 ts_tablet_manager.cc:1398] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:13.641320 8905 tablet_bootstrap.cc:654] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:13.642593 8905 tablet_bootstrap.cc:492] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:13.642720 8905 ts_tablet_manager.cc:1367] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:00:13.643231 8905 raft_consensus.cc:348] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:13.643411 8905 raft_consensus.cc:374] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:13.643469 8905 raft_consensus.cc:729] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:13.643599 8905 consensus_queue.cc:260] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:13.643868 8905 ts_tablet_manager.cc:1398] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:13.644161 8905 tablet_bootstrap.cc:492] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:13.645151 8905 tablet_bootstrap.cc:654] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:13.647030 8905 tablet_bootstrap.cc:492] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd: No bootstrap required, opened a new log
I20240501 08:00:13.647125 8905 ts_tablet_manager.cc:1367] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:00:13.647464 8905 raft_consensus.cc:348] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.647544 8905 raft_consensus.cc:374] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:13.647578 8905 raft_consensus.cc:729] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: FOLLOWER
I20240501 08:00:13.647709 8905 consensus_queue.cc:260] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.647984 8905 ts_tablet_manager.cc:1398] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:13.693156 8880 raft_consensus.cc:482] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:13.693303 8880 raft_consensus.cc:504] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.693732 8880 leader_election.cc:288] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:13.694092 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e44704a9b9c46cf954d8e6937983841" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578" is_pre_election: true
I20240501 08:00:13.694192 7577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e44704a9b9c46cf954d8e6937983841" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd" is_pre_election: true
I20240501 08:00:13.694239 7316 raft_consensus.cc:2454] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 0.
I20240501 08:00:13.694340 7577 raft_consensus.cc:2454] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 0.
I20240501 08:00:13.694586 7646 leader_election.cc:302] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:13.694765 8880 raft_consensus.cc:2781] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:13.694826 8880 raft_consensus.cc:482] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:13.694872 8880 raft_consensus.cc:3037] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.695704 8880 raft_consensus.cc:504] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.696035 8880 leader_election.cc:288] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 election: Requested vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:13.696424 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e44704a9b9c46cf954d8e6937983841" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578"
I20240501 08:00:13.696637 7316 raft_consensus.cc:3037] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.697458 7316 raft_consensus.cc:2454] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 1.
I20240501 08:00:13.697727 7646 leader_election.cc:302] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:13.697707 7577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e44704a9b9c46cf954d8e6937983841" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
I20240501 08:00:13.697835 7577 raft_consensus.cc:3037] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.697923 8880 raft_consensus.cc:2781] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:13.698045 8880 raft_consensus.cc:686] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Becoming Leader. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Running, Role: LEADER
I20240501 08:00:13.698186 8880 consensus_queue.cc:237] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.698699 7577 raft_consensus.cc:2454] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 1.
I20240501 08:00:13.700147 7175 catalog_manager.cc:5484] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09a7b6e2731a474b80411efb62323604 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:13.779860 8772 raft_consensus.cc:482] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:13.780035 8772 raft_consensus.cc:504] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.780612 8772 leader_election.cc:288] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:13.781147 7709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28baea6d7d3f442ba1dbfc5b37de90ac" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604" is_pre_election: true
I20240501 08:00:13.781157 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28baea6d7d3f442ba1dbfc5b37de90ac" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578" is_pre_election: true
I20240501 08:00:13.781313 7709 raft_consensus.cc:2454] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 0.
I20240501 08:00:13.781313 7316 raft_consensus.cc:2454] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 0.
I20240501 08:00:13.781692 7515 leader_election.cc:302] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, 31b02f06be8a4455b94b1254d77005bd; no voters:
I20240501 08:00:13.781877 8772 raft_consensus.cc:2781] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:13.781939 8772 raft_consensus.cc:482] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:13.782016 8772 raft_consensus.cc:3037] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.782866 8772 raft_consensus.cc:504] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.783202 8772 leader_election.cc:288] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 election: Requested vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:13.783588 7709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28baea6d7d3f442ba1dbfc5b37de90ac" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a7b6e2731a474b80411efb62323604"
I20240501 08:00:13.783588 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28baea6d7d3f442ba1dbfc5b37de90ac" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578"
I20240501 08:00:13.783712 7709 raft_consensus.cc:3037] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.783712 7316 raft_consensus.cc:3037] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.784672 7709 raft_consensus.cc:2454] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 1.
I20240501 08:00:13.784672 7316 raft_consensus.cc:2454] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 1.
I20240501 08:00:13.784976 7515 leader_election.cc:302] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, 31b02f06be8a4455b94b1254d77005bd; no voters:
I20240501 08:00:13.785475 8772 raft_consensus.cc:2781] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:13.785558 8772 raft_consensus.cc:686] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Becoming Leader. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Running, Role: LEADER
I20240501 08:00:13.785688 8772 consensus_queue.cc:237] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }
I20240501 08:00:13.787693 7175 catalog_manager.cc:5484] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 31b02f06be8a4455b94b1254d77005bd (127.0.99.131). New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:13.854175 8772 raft_consensus.cc:482] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:13.854305 8772 raft_consensus.cc:504] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:13.854749 8772 leader_election.cc:288] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:13.855113 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4297f95243ac431099a8279c3d9e4b25" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578" is_pre_election: true
I20240501 08:00:13.855180 7448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4297f95243ac431099a8279c3d9e4b25" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" is_pre_election: true
I20240501 08:00:13.855285 7316 raft_consensus.cc:2454] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 0.
I20240501 08:00:13.855301 7448 raft_consensus.cc:2454] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 0.
I20240501 08:00:13.855590 7512 leader_election.cc:302] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:13.855747 8772 raft_consensus.cc:2781] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:13.855805 8772 raft_consensus.cc:482] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:13.855868 8772 raft_consensus.cc:3037] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.856678 8772 raft_consensus.cc:504] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:13.857008 8772 leader_election.cc:288] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 election: Requested vote from peers d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:13.857393 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4297f95243ac431099a8279c3d9e4b25" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f8020266b5455b872a2973fc396578"
I20240501 08:00:13.857393 7448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4297f95243ac431099a8279c3d9e4b25" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
I20240501 08:00:13.857523 7316 raft_consensus.cc:3037] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.857523 7448 raft_consensus.cc:3037] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:13.858531 7316 raft_consensus.cc:2454] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 1.
I20240501 08:00:13.858531 7448 raft_consensus.cc:2454] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 1.
I20240501 08:00:13.858852 7512 leader_election.cc:302] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b02f06be8a4455b94b1254d77005bd, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:13.859025 8772 raft_consensus.cc:2781] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:13.859131 8772 raft_consensus.cc:686] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Becoming Leader. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Running, Role: LEADER
I20240501 08:00:13.859275 8772 consensus_queue.cc:237] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } }
I20240501 08:00:13.861155 7175 catalog_manager.cc:5484] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 31b02f06be8a4455b94b1254d77005bd (127.0.99.131). New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } }
W20240501 08:00:13.983728 8921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:13.983952 8921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:13.986946 8921 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 08:00:14.079941 7316 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:14.080313 7709 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:14.080986 8772 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:14.081131 8840 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:14.083271 7448 raft_consensus.cc:1261] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:14.083745 8840 consensus_queue.cc:1035] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:14.083853 7317 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:14.084348 8678 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:14.085115 7577 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:14.087543 8880 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:14.090505 7315 raft_consensus.cc:1261] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:14.090945 8772 consensus_queue.cc:1035] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 114.166 ms
time per row: 0.0570829 ms
I20240501 08:00:14.151182 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:33901
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:45395
--webserver_port=39227
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:00:14.262046 8975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:14.262287 8975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:00:14.262333 8975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:14.265345 8975 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:00:14.265409 8975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:00:14.265477 8975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 08:00:14.269042 8975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:33901
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=39227
--tserver_master_addrs=127.0.99.190:45395
--never_fsync=true
--heap_profile_path=/tmp/kudu.8975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 08:00:14.269944 8975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:00:14.271106 8975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:14.281284 8983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:14.282222 8980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:14.283449 8975 server_base.cc:1034] running on GCE node
W20240501 08:00:14.284353 8981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:14.284783 8975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:00:14.285393 8975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:00:14.286551 8975 hybrid_clock.cc:648] HybridClock initialized: now 1714550414286524 us; error 42 us; skew 500 ppm
I20240501 08:00:14.288678 8975 webserver.cc:460] Webserver started at http://127.0.99.133:39227/ using document root <none> and password file <none>
I20240501 08:00:14.289254 8975 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:14.289326 8975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:14.293036 8975 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:14.295210 8989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:14.296350 8975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:14.296466 8975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d8aa5ac376a54e1f8417e5af11d28fa7"
format_stamp: "Formatted at 2024-05-01 08:00:04 on dist-test-slave-wslg"
I20240501 08:00:14.296941 8975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:14.320694 8975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:14.321278 8975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:00:14.321424 8975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:14.321904 8975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:14.322795 8975 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:14.322841 8975 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:14.322880 8975 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:14.322894 8975 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:14.331071 8975 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:33901
I20240501 08:00:14.331075 9102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:33901 every 8 connection(s)
I20240501 08:00:14.332212 8975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:00:14.337385 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 8975
I20240501 08:00:14.337776 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:33589
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:45395
--webserver_port=38491
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:00:14.341944 9103 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45395
I20240501 08:00:14.342221 9103 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:14.342746 9103 heartbeater.cc:511] Master 127.0.99.190:45395 requested a full tablet report, sending...
I20240501 08:00:14.344009 7170 ts_manager.cc:194] Re-registered known tserver with Master: d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901)
I20240501 08:00:14.344892 7170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:45683
W20240501 08:00:14.451421 9106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:14.451635 9106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:00:14.451678 9106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:14.454604 9106 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:00:14.454660 9106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:00:14.454725 9106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 08:00:14.458268 9106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:33589
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=38491
--tserver_master_addrs=127.0.99.190:45395
--never_fsync=true
--heap_profile_path=/tmp/kudu.9106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 08:00:14.459138 9106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:00:14.460270 9106 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:14.469421 9114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:14.470208 9112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:14.471412 9111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:14.472337 9106 server_base.cc:1034] running on GCE node
I20240501 08:00:14.472741 9106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:00:14.473321 9106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:00:14.474472 9106 hybrid_clock.cc:648] HybridClock initialized: now 1714550414474445 us; error 45 us; skew 500 ppm
I20240501 08:00:14.476557 9106 webserver.cc:460] Webserver started at http://127.0.99.134:38491/ using document root <none> and password file <none>
I20240501 08:00:14.477200 9106 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:14.477294 9106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:14.481108 9106 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:14.483388 9120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:14.484649 9106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:14.484769 9106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "3b69c5d799e14e438df760453438aa75"
format_stamp: "Formatted at 2024-05-01 08:00:04 on dist-test-slave-wslg"
I20240501 08:00:14.485311 9106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:14.515009 9106 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:14.515625 9106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:00:14.515781 9106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:14.516301 9106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:14.517163 9106 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:14.517217 9106 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:14.517258 9106 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:14.517288 9106 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:14.526353 9106 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:33589
I20240501 08:00:14.526379 9233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:33589 every 8 connection(s)
I20240501 08:00:14.527460 9106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:00:14.534554 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 9106
I20240501 08:00:14.535048 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:41471
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:45395
--webserver_port=33759
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:00:14.536967 9234 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45395
I20240501 08:00:14.537200 9234 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:14.537786 9234 heartbeater.cc:511] Master 127.0.99.190:45395 requested a full tablet report, sending...
I20240501 08:00:14.539417 7170 ts_manager.cc:194] Re-registered known tserver with Master: 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589)
I20240501 08:00:14.540138 7170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:48289
W20240501 08:00:14.655494 9237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:14.655732 9237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:00:14.655803 9237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:14.659235 9237 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:00:14.659319 9237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:00:14.659418 9237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 08:00:14.663334 9237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:41471
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=33759
--tserver_master_addrs=127.0.99.190:45395
--never_fsync=true
--heap_profile_path=/tmp/kudu.9237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 08:00:14.664304 9237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:00:14.665593 9237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:14.674726 9245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:14.675799 9243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:14.676918 9242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:14.677964 9237 server_base.cc:1034] running on GCE node
I20240501 08:00:14.678491 9237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:00:14.679164 9237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:00:14.680362 9237 hybrid_clock.cc:648] HybridClock initialized: now 1714550414680329 us; error 48 us; skew 500 ppm
I20240501 08:00:14.682734 9237 webserver.cc:460] Webserver started at http://127.0.99.135:33759/ using document root <none> and password file <none>
I20240501 08:00:14.683307 9237 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:14.683391 9237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:14.687006 9237 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:00:14.689169 9251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:14.690330 9237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:00:14.690460 9237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "1c8cb9b1059a47cc822ca13a2b931776"
format_stamp: "Formatted at 2024-05-01 08:00:04 on dist-test-slave-wslg"
I20240501 08:00:14.690927 9237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:14.708796 9237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:14.709429 9237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:00:14.709606 9237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:14.710269 9237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:14.711190 9237 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:14.711239 9237 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:14.711297 9237 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:14.711323 9237 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:14.719825 9237 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:41471
I20240501 08:00:14.719846 9364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:41471 every 8 connection(s)
I20240501 08:00:14.720861 9237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:00:14.721578 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 9237
I20240501 08:00:14.722045 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
/tmp/dist-test-taskS8gUPX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:42587
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:45395
--webserver_port=42201
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:00:14.730794 9365 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45395
I20240501 08:00:14.731022 9365 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:14.731580 9365 heartbeater.cc:511] Master 127.0.99.190:45395 requested a full tablet report, sending...
I20240501 08:00:14.732604 7170 ts_manager.cc:194] Re-registered known tserver with Master: 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471)
I20240501 08:00:14.733204 7170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:53499
W20240501 08:00:14.838459 9368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:14.838742 9368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:00:14.838804 9368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:14.842022 9368 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:00:14.842084 9368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:00:14.842177 9368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240501 08:00:14.845733 9368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:42587
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=42201
--tserver_master_addrs=127.0.99.190:45395
--never_fsync=true
--heap_profile_path=/tmp/kudu.9368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 08:00:14.846642 9368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:00:14.847801 9368 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:14.856624 9376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:14.857447 9373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:14.858707 9374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:14.859571 9368 server_base.cc:1034] running on GCE node
I20240501 08:00:14.859998 9368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:00:14.860621 9368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:00:14.861779 9368 hybrid_clock.cc:648] HybridClock initialized: now 1714550414861762 us; error 33 us; skew 500 ppm
I20240501 08:00:14.864014 9368 webserver.cc:460] Webserver started at http://127.0.99.136:42201/ using document root <none> and password file <none>
I20240501 08:00:14.864661 9368 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:14.864733 9368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:14.868254 9368 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:00:14.870445 9382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:14.871639 9368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:14.871757 9368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "1807c0c2e7d04b20b0cda311c42eef8e"
format_stamp: "Formatted at 2024-05-01 08:00:04 on dist-test-slave-wslg"
I20240501 08:00:14.872237 9368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:14.896876 9368 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:14.897518 9368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:00:14.897675 9368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:14.898267 9368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:14.899158 9368 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:14.899207 9368 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:14.899266 9368 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:14.899291 9368 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:14.908087 9368 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:42587
I20240501 08:00:14.908097 9495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:42587 every 8 connection(s)
I20240501 08:00:14.909245 9368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:00:14.917618 9496 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45395
I20240501 08:00:14.917843 9496 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:14.918437 9496 heartbeater.cc:511] Master 127.0.99.190:45395 requested a full tablet report, sending...
I20240501 08:00:14.919178 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu as pid 9368
I20240501 08:00:14.919524 7170 ts_manager.cc:194] Re-registered known tserver with Master: 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587)
I20240501 08:00:14.920204 7170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:36327
I20240501 08:00:15.281181 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.283185 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.283502 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:15.286691 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:15.307901 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.315652 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.318413 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.318444 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.318512 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.319188 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.320956 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:15.321400 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:15.337147 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.338239 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.339713 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.346956 9103 heartbeater.cc:503] Master 127.0.99.190:45395 was elected leader, sending a full tablet report...
I20240501 08:00:15.351898 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.514859 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.518908 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.529508 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.538995 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:15.539671 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.542022 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.542682 9234 heartbeater.cc:503] Master 127.0.99.190:45395 was elected leader, sending a full tablet report...
I20240501 08:00:15.546553 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.552165 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.562215 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:15.566181 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:15.574451 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.578224 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.582158 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.595613 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.597934 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.603623 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.609088 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.617084 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:15.624469 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.633827 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.635545 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:15.637189 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.645104 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.663712 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:15.735435 9365 heartbeater.cc:503] Master 127.0.99.190:45395 was elected leader, sending a full tablet report...
I20240501 08:00:15.784232 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.797750 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.811844 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.825901 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.837299 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.838366 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:15.841275 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:15.846325 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.879143 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.887694 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:15.892313 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.895125 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.898977 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.922273 9496 heartbeater.cc:503] Master 127.0.99.190:45395 was elected leader, sending a full tablet report...
I20240501 08:00:15.929358 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:15.939131 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:15.944614 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.957931 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.965929 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:15.976940 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.999058 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.004951 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:16.006306 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.021154 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.039062 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.091420 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.102449 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.105412 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:16.115718 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.122794 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.125224 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:16.126677 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.144250 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.154152 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.166770 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.170146 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:16.188077 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.216157 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.216576 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:16.238029 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.239324 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.269863 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.275300 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.278471 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.282487 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.288591 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.293329 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:16.311120 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:16.319479 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.372155 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.373099 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.380012 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.382870 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.385408 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.386137 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.406414 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:16.413089 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:16.450745 7447 consensus_queue.cc:237] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:16.453217 7577 raft_consensus.cc:1261] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.453814 8819 consensus_queue.cc:1035] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.454176 7709 raft_consensus.cc:1261] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.455338 8819 consensus_queue.cc:1035] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.457517 8819 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:16.461650 7168 catalog_manager.cc:5484] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:00:16.462008 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54ced80b9ff943ec88c27b7842500b3d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.463325 7577 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:16.467118 7709 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:16.477602 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.481180 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.483685 7577 consensus_queue.cc:237] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:16.486518 7315 raft_consensus.cc:1261] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.486896 9594 consensus_queue.cc:1035] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.488698 7447 raft_consensus.cc:1261] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.489055 9594 consensus_queue.cc:1035] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:00:16.497659 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7377810e08104526b8b8acdf886a3368. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.506513 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.513031 9594 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:16.515858 7447 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:16.517537 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.518280 7315 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:16.521412 7167 catalog_manager.cc:5484] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:16.525835 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:16.527137 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:16.528095 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.558136 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.565058 7577 consensus_queue.cc:237] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:16.567580 7709 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.568059 8772 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.570343 7447 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.571220 8772 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:00:16.574988 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21d76484f5dd499fb6fc3d534bffb779. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.578155 8772 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:16.578392 7447 consensus_queue.cc:237] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:16.580807 7448 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:16.581705 7709 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:16.582160 7172 catalog_manager.cc:5484] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:16.585863 7577 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.586356 7315 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.586817 10147 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.586987 10146 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:00:16.592238 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3abb5202d8d24d9b9a67f7a87b7300a6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.602288 10147 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:16.605571 7315 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:16.606261 7170 catalog_manager.cc:5484] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:16.614727 7577 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:16.645743 7315 consensus_queue.cc:237] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:16.647922 7709 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.648160 7447 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.648581 10080 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.648802 9528 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.653963 10080 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:16.655767 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.656868 7709 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:16.658769 7168 catalog_manager.cc:5484] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:16.664958 7447 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:16.671876 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:16.679515 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5970772c02d04e0b95a2f2feb7dcd655. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.684854 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.686234 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.689702 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.695406 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:16.701915 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.704602 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:16.708608 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.730125 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.742684 7315 consensus_queue.cc:237] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:16.744343 7577 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.744762 10264 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.744830 7447 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.745364 10264 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.747377 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.748172 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.750047 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:16.755898 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e178f7c3c17c4e1ca3d16b6441f76e77. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.761034 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:16.772478 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.773607 10080 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:16.775251 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:16.776829 7168 catalog_manager.cc:5484] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:16.778165 7577 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:16.779196 7447 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:16.791335 7577 consensus_queue.cc:237] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:16.793327 7447 consensus_queue.cc:237] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:16.801266 7315 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:16.801919 10147 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 08:00:16.802776 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3abb5202d8d24d9b9a67f7a87b7300a6. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:16.804380 7382 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3abb5202d8d24d9b9a67f7a87b7300a6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.804245 10147 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
W20240501 08:00:16.805812 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21d76484f5dd499fb6fc3d534bffb779. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.806172 7315 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:16.806424 7447 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:16.806489 7709 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:16.806885 10185 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:16.807379 10185 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:16.808558 7167 catalog_manager.cc:5484] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:16.808961 7579 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:16.811163 8819 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 08:00:16.811928 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21d76484f5dd499fb6fc3d534bffb779. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.811822 8772 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:16.812494 7579 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:16.814148 7709 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:16.816834 7168 catalog_manager.cc:5484] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:16.819089 7447 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:16.836961 7709 consensus_queue.cc:237] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:16.839005 7579 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.839058 7447 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.839656 8678 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.840046 8678 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.842420 7577 consensus_queue.cc:237] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:16.844087 7315 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.844640 10185 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.846186 7448 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:00:16.846598 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bde42e485c4b49ce8acd694a25ba683a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.846709 10185 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:00:16.848098 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81ccdd855b584d65860fd7a5777920a3. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.849102 10302 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:16.851804 7448 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:16.856859 7167 catalog_manager.cc:5484] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:16.862496 10185 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:16.864646 7447 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:16.865470 7315 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:16.867282 7168 catalog_manager.cc:5484] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:16.871989 7709 consensus_queue.cc:237] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:16.873955 7315 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.874581 7579 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.875725 10323 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.877524 7579 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:16.877691 10335 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.883262 10335 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
W20240501 08:00:16.883459 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e44704a9b9c46cf954d8e6937983841. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.885253 7315 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:16.885422 7577 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:16.886446 7167 catalog_manager.cc:5484] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:16.895699 7447 consensus_queue.cc:237] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:16.898483 7577 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:16.898890 10147 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:00:16.899086 7382 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3abb5202d8d24d9b9a67f7a87b7300a6. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:16.899307 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3abb5202d8d24d9b9a67f7a87b7300a6. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:16.899574 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3abb5202d8d24d9b9a67f7a87b7300a6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.900565 7315 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:16.901007 10147 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:16.901409 10146 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:16.901743 7448 consensus_queue.cc:237] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:16.904675 7579 raft_consensus.cc:1261] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.904851 7709 raft_consensus.cc:1261] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.904816 7171 catalog_manager.cc:5484] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 5 to 6, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:16.905527 8819 consensus_queue.cc:1035] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.905771 7577 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:16.906872 10147 consensus_queue.cc:1035] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.907929 7315 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:16.910364 10386 ts_tablet_manager.cc:902] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
W20240501 08:00:16.911525 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f647cfc562794e65a05918b241014d72. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.914516 10386 tablet_copy_client.cc:308] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:16.924038 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet 7377810e08104526b8b8acdf886a3368 from peer 1807c0c2e7d04b20b0cda311c42eef8e ({username='slave'} at 127.0.99.136:42359)
I20240501 08:00:16.924222 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet 7377810e08104526b8b8acdf886a3368 from peer 1807c0c2e7d04b20b0cda311c42eef8e at {username='slave'} at 127.0.99.136:42359: session id = 1807c0c2e7d04b20b0cda311c42eef8e-7377810e08104526b8b8acdf886a3368
I20240501 08:00:16.925841 7599 tablet_copy_source_session.cc:205] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:16.930814 10386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7377810e08104526b8b8acdf886a3368. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:16.934979 7579 consensus_queue.cc:237] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
W20240501 08:00:16.936651 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc6f7dcc930e4937865d37cb3c3bb3d2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.936718 7315 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.936821 7708 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.937124 8772 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.937325 10186 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.938222 10381 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:16.939921 7709 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:16.940775 7579 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:16.943101 7171 catalog_manager.cc:5484] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:16.949295 7709 consensus_queue.cc:237] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:16.951323 8772 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
W20240501 08:00:16.952405 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e44704a9b9c46cf954d8e6937983841. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.954057 7315 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:16.954558 10335 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:16.956364 7709 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:16.957315 7315 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:16.957720 7171 catalog_manager.cc:5484] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:16.958544 10335 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:16.961696 7709 consensus_queue.cc:237] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
W20240501 08:00:16.961967 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e44704a9b9c46cf954d8e6937983841. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.962877 7579 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 08:00:16.964426 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dde9710a5eb4baa881d9dd04affe184. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:16.965741 10323 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:16.966298 7315 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:16.966975 7168 catalog_manager.cc:5484] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:16.968452 7316 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.968730 7577 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:16.968995 10419 ts_tablet_manager.cc:902] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7: Initiating tablet copy from peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:16.969805 10419 tablet_copy_client.cc:308] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40707
I20240501 08:00:16.970561 10323 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.971640 10386 tablet_copy_client.cc:783] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:16.971879 10386 tablet_copy_client.cc:647] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:16.973605 7579 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:16.974222 8678 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:16.974426 10386 tablet_copy_client.cc:523] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:16.976958 10386 tablet_bootstrap.cc:492] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap starting.
I20240501 08:00:16.980754 7468 tablet_copy_service.cc:140] P d6b4be7bbd1649d0b7c5e8efe81f759f: Received BeginTabletCopySession request for tablet 54ced80b9ff943ec88c27b7842500b3d from peer d8aa5ac376a54e1f8417e5af11d28fa7 ({username='slave'} at 127.0.99.133:43147)
I20240501 08:00:16.980916 7468 tablet_copy_service.cc:161] P d6b4be7bbd1649d0b7c5e8efe81f759f: Beginning new tablet copy session on tablet 54ced80b9ff943ec88c27b7842500b3d from peer d8aa5ac376a54e1f8417e5af11d28fa7 at {username='slave'} at 127.0.99.133:43147: session id = d8aa5ac376a54e1f8417e5af11d28fa7-54ced80b9ff943ec88c27b7842500b3d
I20240501 08:00:16.982085 8678 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:16.982439 7468 tablet_copy_source_session.cc:205] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:16.984481 7317 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:16.986224 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.986399 10386 log.cc:824] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e: Log is configured to *not* fsync() on all Append() calls
I20240501 08:00:16.987514 7171 catalog_manager.cc:5484] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:16.990124 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.991837 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54ced80b9ff943ec88c27b7842500b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:16.993356 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:16.999352 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.999675 10419 tablet_copy_client.cc:783] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:16.999886 10419 tablet_copy_client.cc:647] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.002117 10419 tablet_copy_client.cc:523] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.004120 10419 tablet_bootstrap.cc:492] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap starting.
I20240501 08:00:17.011196 10419 log.cc:824] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7: Log is configured to *not* fsync() on all Append() calls
I20240501 08:00:17.022926 7579 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:17.030305 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.031070 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:17.047369 10419 tablet_bootstrap.cc:492] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:17.047822 10419 tablet_bootstrap.cc:492] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap complete.
I20240501 08:00:17.048179 10419 ts_tablet_manager.cc:1367] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent bootstrapping tablet: real 0.044s user 0.037s sys 0.000s
I20240501 08:00:17.051147 10419 raft_consensus.cc:348] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:17.051451 10419 raft_consensus.cc:729] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8aa5ac376a54e1f8417e5af11d28fa7, State: Initialized, Role: LEARNER
I20240501 08:00:17.051823 10419 consensus_queue.cc:260] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:17.053866 10419 ts_tablet_manager.cc:1398] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 08:00:17.054687 7468 tablet_copy_service.cc:342] P d6b4be7bbd1649d0b7c5e8efe81f759f: Request end of tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-54ced80b9ff943ec88c27b7842500b3d received from {username='slave'} at 127.0.99.133:43147
I20240501 08:00:17.054785 7468 tablet_copy_service.cc:433] P d6b4be7bbd1649d0b7c5e8efe81f759f: ending tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-54ced80b9ff943ec88c27b7842500b3d on tablet 54ced80b9ff943ec88c27b7842500b3d with peer d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:17.058439 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:17.059583 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:17.061703 7709 consensus_queue.cc:237] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:17.065670 7447 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 08:00:17.066100 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81ccdd855b584d65860fd7a5777920a3. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.066762 7579 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:17.067813 10323 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:17.068058 10323 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:17.071487 8678 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.074061 7447 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.074335 7167 catalog_manager.cc:5484] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:17.075958 7579 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
W20240501 08:00:17.077250 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81ccdd855b584d65860fd7a5777920a3. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.091187 7315 consensus_queue.cc:237] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:00:17.093246 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1088a116ccec4020aec562a69791a188. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.093660 7579 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.094043 10264 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.095552 7447 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.096024 10264 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.099520 10264 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.099726 7315 consensus_queue.cc:237] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:17.101187 10386 tablet_bootstrap.cc:492] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:17.101270 7579 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.101555 10386 tablet_bootstrap.cc:492] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap complete.
I20240501 08:00:17.101864 10386 ts_tablet_manager.cc:1367] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent bootstrapping tablet: real 0.125s user 0.037s sys 0.002s
I20240501 08:00:17.102540 7448 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.104070 7709 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.105149 7171 catalog_manager.cc:5484] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:17.105373 10386 raft_consensus.cc:348] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:17.105621 10386 raft_consensus.cc:729] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1807c0c2e7d04b20b0cda311c42eef8e, State: Initialized, Role: LEARNER
I20240501 08:00:17.105966 10386 consensus_queue.cc:260] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:17.106349 10306 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.106899 10386 ts_tablet_manager.cc:1398] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240501 08:00:17.107663 10306 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.109676 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-7377810e08104526b8b8acdf886a3368 received from {username='slave'} at 127.0.99.136:42359
I20240501 08:00:17.109766 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-7377810e08104526b8b8acdf886a3368 on tablet 7377810e08104526b8b8acdf886a3368 with peer 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:17.110419 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d8ae3c82c9d4c5c858fa3f9d5959eb7. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.112613 7447 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.113705 10306 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.115975 7447 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.116417 7708 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.119334 7169 catalog_manager.cc:5484] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.128815 7579 consensus_queue.cc:237] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:17.130656 7447 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 08:00:17.130818 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bde42e485c4b49ce8acd694a25ba683a. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:17.130935 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bde42e485c4b49ce8acd694a25ba683a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.131088 7315 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:17.131479 10185 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:17.131624 10186 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:17.133436 10185 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.136281 7167 catalog_manager.cc:5484] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 4 to 5, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:17.137231 7315 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.138049 10499 ts_tablet_manager.cc:902] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:17.139861 7447 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.142066 7579 consensus_queue.cc:237] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:17.143622 7708 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:17.143980 10186 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:17.146247 9594 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
W20240501 08:00:17.146843 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc6f7dcc930e4937865d37cb3c3bb3d2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.148443 7708 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
W20240501 08:00:17.151625 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc6f7dcc930e4937865d37cb3c3bb3d2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.151845 7172 catalog_manager.cc:5484] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 4 to 5, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:17.152889 7315 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:17.153260 10185 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:17.157277 7317 consensus_queue.cc:237] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:17.157618 7315 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:17.157963 10499 tablet_copy_client.cc:308] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:17.159353 7447 raft_consensus.cc:1261] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:00:17.159617 7251 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8fdc2b9c2ef44b0bfec59a1a2f6b626. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.159785 9528 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.160135 7708 raft_consensus.cc:1261] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.160522 9528 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.166960 10264 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.167471 7447 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.169720 7708 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.172518 7167 catalog_manager.cc:5484] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.182686 7708 consensus_queue.cc:237] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:17.183437 7579 consensus_queue.cc:237] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
W20240501 08:00:17.185631 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21d76484f5dd499fb6fc3d534bffb779. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.185817 7447 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 08:00:17.186051 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dde9710a5eb4baa881d9dd04affe184. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.186380 8772 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:00:17.186741 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dde9710a5eb4baa881d9dd04affe184. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:17.186740 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21d76484f5dd499fb6fc3d534bffb779. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:17.186873 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21d76484f5dd499fb6fc3d534bffb779. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.187008 7315 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:17.187187 7579 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:17.187568 8678 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:17.187951 10302 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:17.187578 7708 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:17.190511 8772 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:17.190464 10302 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.190765 7579 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.190984 8772 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.194553 7169 catalog_manager.cc:5484] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:17.194927 7169 catalog_manager.cc:5484] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.198999 7315 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.200860 7447 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.201042 7448 consensus_queue.cc:237] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:17.203590 9057 raft_consensus.cc:1261] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:17.204316 7708 raft_consensus.cc:1261] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:17.204723 10384 consensus_queue.cc:1035] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:17.205217 7579 raft_consensus.cc:1261] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:17.205698 10384 consensus_queue.cc:1035] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:17.205912 10146 consensus_queue.cc:1035] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 08:00:17.206071 7381 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54ced80b9ff943ec88c27b7842500b3d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.208490 10146 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.208948 7579 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.211723 10147 raft_consensus.cc:1050] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f: attempting to promote NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 to VOTER
I20240501 08:00:17.212182 10147 consensus_queue.cc:237] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:17.214607 9056 raft_consensus.cc:1261] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 08:00:17.214848 7381 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54ced80b9ff943ec88c27b7842500b3d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.215173 10384 consensus_queue.cc:1035] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:17.215256 9057 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.215667 7708 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.216533 7171 catalog_manager.cc:5484] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:17.217183 7709 raft_consensus.cc:1261] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:17.218791 7577 raft_consensus.cc:1261] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:17.219403 10384 consensus_queue.cc:1035] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:17.219724 10384 consensus_queue.cc:1035] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:17.222175 7708 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.223346 10146 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.223704 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet 5970772c02d04e0b95a2f2feb7dcd655 from peer 1c8cb9b1059a47cc822ca13a2b931776 ({username='slave'} at 127.0.99.135:39313)
I20240501 08:00:17.223845 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet 5970772c02d04e0b95a2f2feb7dcd655 from peer 1c8cb9b1059a47cc822ca13a2b931776 at {username='slave'} at 127.0.99.135:39313: session id = 1c8cb9b1059a47cc822ca13a2b931776-5970772c02d04e0b95a2f2feb7dcd655
I20240501 08:00:17.225450 7336 tablet_copy_source_session.cc:205] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.226563 7708 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.227540 7172 catalog_manager.cc:5484] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:17.228312 9056 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.229625 10499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5970772c02d04e0b95a2f2feb7dcd655. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.230230 7577 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.233192 7447 consensus_queue.cc:237] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:17.234958 7709 raft_consensus.cc:1261] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:17.235525 10151 consensus_queue.cc:1035] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:17.236272 9056 raft_consensus.cc:1261] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:17.236295 7577 raft_consensus.cc:1261] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:17.236865 10151 consensus_queue.cc:1035] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:17.237156 10151 consensus_queue.cc:1035] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:17.238541 7578 consensus_queue.cc:237] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:17.238703 7577 consensus_queue.cc:237] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:17.239415 10499 tablet_copy_client.cc:783] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.239617 10499 tablet_copy_client.cc:647] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.239711 10146 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
W20240501 08:00:17.241098 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bde42e485c4b49ce8acd694a25ba683a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.241123 7317 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.241123 7315 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:17.241504 7709 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.241925 8772 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.242056 9594 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:17.242288 10185 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.242795 7447 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 08:00:17.242895 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bde42e485c4b49ce8acd694a25ba683a. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:17.243240 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bde42e485c4b49ce8acd694a25ba683a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.244179 10185 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.244518 10186 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:17.244730 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 54ced80b9ff943ec88c27b7842500b3d with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:17.245294 7315 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.246048 10499 tablet_copy_client.cc:523] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.246325 7169 catalog_manager.cc:5484] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 6 to 7, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:17.248030 7577 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:17.248152 7171 catalog_manager.cc:5484] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
W20240501 08:00:17.248613 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8311d0921e9e417090e441884615525a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.249450 10499 tablet_bootstrap.cc:492] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap starting.
I20240501 08:00:17.250391 7447 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.251219 9056 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:17.255457 7710 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:17.256248 10532 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:17.256798 7709 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:17.259585 7317 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:17.261705 7173 catalog_manager.cc:5484] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:17.261865 7448 consensus_queue.cc:237] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:17.266297 7709 raft_consensus.cc:1261] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:17.267658 10146 consensus_queue.cc:1035] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:17.269496 10151 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:17.269915 7709 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:17.271188 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 54ced80b9ff943ec88c27b7842500b3d with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:17.272881 7171 catalog_manager.cc:5484] T 54ced80b9ff943ec88c27b7842500b3d P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 7 to 8, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:17.274544 9056 raft_consensus.cc:1261] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:17.274574 10499 log.cc:824] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776: Log is configured to *not* fsync() on all Append() calls
I20240501 08:00:17.277339 7558 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ced80b9ff943ec88c27b7842500b3d with delete_type TABLET_DATA_TOMBSTONED (TS 31b02f06be8a4455b94b1254d77005bd not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:33196
I20240501 08:00:17.277937 10384 consensus_queue.cc:1035] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:17.280431 9056 raft_consensus.cc:2932] T 54ced80b9ff943ec88c27b7842500b3d P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:17.281147 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:17.281960 7317 consensus_queue.cc:237] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:17.283802 7709 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.284169 9528 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.284332 7448 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:00:17.284449 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02b42b5fd3d440ea950b35ed461b8e12. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.285141 9528 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.285600 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.291236 9528 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:17.294185 7172 catalog_manager.cc:5484] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:17.295086 7448 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:17.298449 7317 consensus_queue.cc:237] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:00:17.300231 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e42ef8df4884bc9bc33f5ed4c164c8c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.300974 7710 raft_consensus.cc:1261] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.301301 7579 raft_consensus.cc:1261] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.301388 10306 consensus_queue.cc:1035] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.302742 10597 ts_tablet_manager.cc:902] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75: Initiating tablet copy from peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:17.303496 10306 consensus_queue.cc:1035] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.303653 10597 tablet_copy_client.cc:308] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40707
I20240501 08:00:17.306257 7709 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:17.319892 10264 raft_consensus.cc:2932] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.320500 10615 ts_tablet_manager.cc:902] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776: Initiating tablet copy from peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:17.321089 10615 tablet_copy_client.cc:308] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40707
I20240501 08:00:17.321091 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:17.325218 7579 raft_consensus.cc:2932] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.328692 7468 tablet_copy_service.cc:140] P d6b4be7bbd1649d0b7c5e8efe81f759f: Received BeginTabletCopySession request for tablet 3abb5202d8d24d9b9a67f7a87b7300a6 from peer 1c8cb9b1059a47cc822ca13a2b931776 ({username='slave'} at 127.0.99.135:41949)
I20240501 08:00:17.328797 7468 tablet_copy_service.cc:161] P d6b4be7bbd1649d0b7c5e8efe81f759f: Beginning new tablet copy session on tablet 3abb5202d8d24d9b9a67f7a87b7300a6 from peer 1c8cb9b1059a47cc822ca13a2b931776 at {username='slave'} at 127.0.99.135:41949: session id = 1c8cb9b1059a47cc822ca13a2b931776-3abb5202d8d24d9b9a67f7a87b7300a6
I20240501 08:00:17.329743 7468 tablet_copy_source_session.cc:205] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.330054 7171 catalog_manager.cc:5484] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:17.331205 10386 ts_tablet_manager.cc:902] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e: Initiating tablet copy from peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:17.331626 10386 tablet_copy_client.cc:308] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40707
I20240501 08:00:17.332379 10419 ts_tablet_manager.cc:902] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7: Initiating tablet copy from peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:17.332780 10419 tablet_copy_client.cc:308] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40707
I20240501 08:00:17.334067 7468 tablet_copy_service.cc:140] P d6b4be7bbd1649d0b7c5e8efe81f759f: Received BeginTabletCopySession request for tablet 3abb5202d8d24d9b9a67f7a87b7300a6 from peer d8aa5ac376a54e1f8417e5af11d28fa7 ({username='slave'} at 127.0.99.133:43147)
I20240501 08:00:17.334159 7468 tablet_copy_service.cc:161] P d6b4be7bbd1649d0b7c5e8efe81f759f: Beginning new tablet copy session on tablet 3abb5202d8d24d9b9a67f7a87b7300a6 from peer d8aa5ac376a54e1f8417e5af11d28fa7 at {username='slave'} at 127.0.99.133:43147: session id = d8aa5ac376a54e1f8417e5af11d28fa7-3abb5202d8d24d9b9a67f7a87b7300a6
I20240501 08:00:17.335925 7468 tablet_copy_source_session.cc:205] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.339565 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3abb5202d8d24d9b9a67f7a87b7300a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.341672 10419 tablet_copy_client.cc:783] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.341806 10419 tablet_copy_client.cc:647] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.346118 7468 tablet_copy_service.cc:140] P d6b4be7bbd1649d0b7c5e8efe81f759f: Received BeginTabletCopySession request for tablet 3abb5202d8d24d9b9a67f7a87b7300a6 from peer 1807c0c2e7d04b20b0cda311c42eef8e ({username='slave'} at 127.0.99.136:53469)
I20240501 08:00:17.346307 7468 tablet_copy_service.cc:161] P d6b4be7bbd1649d0b7c5e8efe81f759f: Beginning new tablet copy session on tablet 3abb5202d8d24d9b9a67f7a87b7300a6 from peer 1807c0c2e7d04b20b0cda311c42eef8e at {username='slave'} at 127.0.99.136:53469: session id = 1807c0c2e7d04b20b0cda311c42eef8e-3abb5202d8d24d9b9a67f7a87b7300a6
I20240501 08:00:17.347332 7468 tablet_copy_source_session.cc:205] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.348181 10386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3abb5202d8d24d9b9a67f7a87b7300a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.351223 10386 tablet_copy_client.cc:783] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.351390 10386 tablet_copy_client.cc:647] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.352368 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:17.352943 10615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3abb5202d8d24d9b9a67f7a87b7300a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.354962 10386 tablet_copy_client.cc:523] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.358398 10615 tablet_copy_client.cc:783] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.358608 10615 tablet_copy_client.cc:647] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.358929 10419 tablet_copy_client.cc:523] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.359247 10386 tablet_bootstrap.cc:492] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap starting.
I20240501 08:00:17.360224 10419 tablet_bootstrap.cc:492] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap starting.
I20240501 08:00:17.366291 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:17.366405 10579 tablet_replica.cc:330] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd: stopping tablet replica
I20240501 08:00:17.366762 10579 raft_consensus.cc:2229] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:17.368222 10579 raft_consensus.cc:2258] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:17.370833 7467 tablet_copy_service.cc:140] P d6b4be7bbd1649d0b7c5e8efe81f759f: Received BeginTabletCopySession request for tablet f647cfc562794e65a05918b241014d72 from peer 3b69c5d799e14e438df760453438aa75 ({username='slave'} at 127.0.99.134:55743)
I20240501 08:00:17.370937 7467 tablet_copy_service.cc:161] P d6b4be7bbd1649d0b7c5e8efe81f759f: Beginning new tablet copy session on tablet f647cfc562794e65a05918b241014d72 from peer 3b69c5d799e14e438df760453438aa75 at {username='slave'} at 127.0.99.134:55743: session id = 3b69c5d799e14e438df760453438aa75-f647cfc562794e65a05918b241014d72
I20240501 08:00:17.371876 7467 tablet_copy_source_session.cc:205] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.375803 10597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f647cfc562794e65a05918b241014d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.385625 7710 raft_consensus.cc:2932] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.393113 10597 tablet_copy_client.cc:783] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.393414 10597 tablet_copy_client.cc:647] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.393623 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:17.397738 10597 tablet_copy_client.cc:523] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.398834 10641 ts_tablet_manager.cc:902] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:17.399433 10641 tablet_copy_client.cc:308] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:17.400744 10597 tablet_bootstrap.cc:492] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75: Bootstrap starting.
I20240501 08:00:17.406881 10638 ts_tablet_manager.cc:902] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:17.406888 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet e178f7c3c17c4e1ca3d16b6441f76e77 from peer d8aa5ac376a54e1f8417e5af11d28fa7 ({username='slave'} at 127.0.99.133:50977)
I20240501 08:00:17.406988 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet e178f7c3c17c4e1ca3d16b6441f76e77 from peer d8aa5ac376a54e1f8417e5af11d28fa7 at {username='slave'} at 127.0.99.133:50977: session id = d8aa5ac376a54e1f8417e5af11d28fa7-e178f7c3c17c4e1ca3d16b6441f76e77
I20240501 08:00:17.407474 10638 tablet_copy_client.cc:308] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:00:17.408041 7336 tablet_copy_source_session.cc:205] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.409416 10641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e178f7c3c17c4e1ca3d16b6441f76e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.412111 10641 tablet_copy_client.cc:783] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.412065 7710 consensus_queue.cc:237] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:17.412284 10641 tablet_copy_client.cc:647] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 1 WAL segments...
W20240501 08:00:17.413821 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a37e631ae4f46a2be1ad5c43db5724b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.414394 7579 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.414920 8678 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.415668 7448 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.417598 8678 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.417904 8678 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.421123 7174 catalog_manager.cc:5484] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:17.421275 10597 log.cc:824] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75: Log is configured to *not* fsync() on all Append() calls
I20240501 08:00:17.422412 7579 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.424832 10386 tablet_bootstrap.cc:492] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:17.425918 10386 tablet_bootstrap.cc:492] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap complete.
I20240501 08:00:17.426138 10386 ts_tablet_manager.cc:1367] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent bootstrapping tablet: real 0.067s user 0.026s sys 0.005s
I20240501 08:00:17.426833 10386 raft_consensus.cc:348] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:17.426945 10386 raft_consensus.cc:729] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1807c0c2e7d04b20b0cda311c42eef8e, State: Initialized, Role: LEARNER
I20240501 08:00:17.427052 10386 consensus_queue.cc:260] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:17.427187 7448 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.427846 10386 ts_tablet_manager.cc:1398] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:17.428326 7467 tablet_copy_service.cc:342] P d6b4be7bbd1649d0b7c5e8efe81f759f: Request end of tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-3abb5202d8d24d9b9a67f7a87b7300a6 received from {username='slave'} at 127.0.99.136:53469
I20240501 08:00:17.428393 7467 tablet_copy_service.cc:433] P d6b4be7bbd1649d0b7c5e8efe81f759f: ending tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-3abb5202d8d24d9b9a67f7a87b7300a6 on tablet 3abb5202d8d24d9b9a67f7a87b7300a6 with peer 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:17.431269 10641 tablet_copy_client.cc:523] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.432598 10641 tablet_bootstrap.cc:492] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap starting.
I20240501 08:00:17.437080 7730 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 1e44704a9b9c46cf954d8e6937983841 from peer d8aa5ac376a54e1f8417e5af11d28fa7 ({username='slave'} at 127.0.99.133:35511)
I20240501 08:00:17.437237 7730 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 1e44704a9b9c46cf954d8e6937983841 from peer d8aa5ac376a54e1f8417e5af11d28fa7 at {username='slave'} at 127.0.99.133:35511: session id = d8aa5ac376a54e1f8417e5af11d28fa7-1e44704a9b9c46cf954d8e6937983841
I20240501 08:00:17.438660 7730 tablet_copy_source_session.cc:205] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.439803 10638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e44704a9b9c46cf954d8e6937983841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.441741 10638 tablet_copy_client.cc:783] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.441900 10638 tablet_copy_client.cc:647] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.443387 10638 tablet_copy_client.cc:523] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.444581 10638 tablet_bootstrap.cc:492] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap starting.
I20240501 08:00:17.444649 10615 tablet_copy_client.cc:523] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.445103 7448 consensus_queue.cc:237] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:17.446142 10615 tablet_bootstrap.cc:492] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap starting.
I20240501 08:00:17.446950 10579 ts_tablet_manager.cc:1875] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:17.447149 9449 raft_consensus.cc:1203] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 08:00:17.448374 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd17760cd5c2454db97610b2f71c0ee2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.450415 10579 ts_tablet_manager.cc:1888] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:00:17.450510 10579 log.cc:1196] T 54ced80b9ff943ec88c27b7842500b3d P 31b02f06be8a4455b94b1254d77005bd: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/54ced80b9ff943ec88c27b7842500b3d
I20240501 08:00:17.452119 7159 catalog_manager.cc:4830] TS 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861): tablet 54ced80b9ff943ec88c27b7842500b3d (table rebalance_test_table_1 [id=ba8a985906c04117a440386d3011b936]) successfully deleted
I20240501 08:00:17.458802 7579 raft_consensus.cc:1261] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.459023 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:17.459259 10384 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.464107 9298 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ced80b9ff943ec88c27b7842500b3d with delete_type TABLET_DATA_TOMBSTONED (TS 1c8cb9b1059a47cc822ca13a2b931776 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:40266
I20240501 08:00:17.464433 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.464828 7710 raft_consensus.cc:1261] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.466050 10384 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.466533 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.471313 10386 ts_tablet_manager.cc:902] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:17.471757 10386 tablet_copy_client.cc:308] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:00:17.480599 7709 consensus_queue.cc:237] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:17.484170 7577 raft_consensus.cc:1261] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.484645 10335 consensus_queue.cc:1035] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.486528 7317 raft_consensus.cc:1261] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.487154 10335 consensus_queue.cc:1035] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:00:17.490244 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df6dd3d8482d4931b1d3014a67e739c6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.491330 7730 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 1e44704a9b9c46cf954d8e6937983841 from peer 1807c0c2e7d04b20b0cda311c42eef8e ({username='slave'} at 127.0.99.136:56099)
I20240501 08:00:17.491439 7730 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 1e44704a9b9c46cf954d8e6937983841 from peer 1807c0c2e7d04b20b0cda311c42eef8e at {username='slave'} at 127.0.99.136:56099: session id = 1807c0c2e7d04b20b0cda311c42eef8e-1e44704a9b9c46cf954d8e6937983841
I20240501 08:00:17.492362 7730 tablet_copy_source_session.cc:205] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.493240 10386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e44704a9b9c46cf954d8e6937983841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.496924 10386 tablet_copy_client.cc:783] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.497143 10386 tablet_copy_client.cc:647] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.503616 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:17.509354 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:17.517958 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.520963 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:17.522265 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:17.531177 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:17.553514 10687 ts_tablet_manager.cc:902] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:17.554168 10687 tablet_copy_client.cc:308] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:00:17.559229 7730 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 81ccdd855b584d65860fd7a5777920a3 from peer 1c8cb9b1059a47cc822ca13a2b931776 ({username='slave'} at 127.0.99.135:43955)
I20240501 08:00:17.559361 7730 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 81ccdd855b584d65860fd7a5777920a3 from peer 1c8cb9b1059a47cc822ca13a2b931776 at {username='slave'} at 127.0.99.135:43955: session id = 1c8cb9b1059a47cc822ca13a2b931776-81ccdd855b584d65860fd7a5777920a3
I20240501 08:00:17.560319 7730 tablet_copy_source_session.cc:205] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.571247 10323 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.571796 7577 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.573310 7317 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:17.574386 7172 catalog_manager.cc:5484] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:00:17.580322 7159 catalog_manager.cc:4797] TS 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): delete failed for tablet 54ced80b9ff943ec88c27b7842500b3d because the tablet was not found. No further retry: Not found: Tablet not found: 54ced80b9ff943ec88c27b7842500b3d
I20240501 08:00:17.587740 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:17.587864 10384 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.591143 7577 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.594457 7174 catalog_manager.cc:5484] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.597815 7710 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.603142 10717 ts_tablet_manager.cc:902] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:17.604193 10717 tablet_copy_client.cc:308] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:17.604468 10687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ccdd855b584d65860fd7a5777920a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.610787 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7 from peer 3b69c5d799e14e438df760453438aa75 ({username='slave'} at 127.0.99.134:33617)
I20240501 08:00:17.610903 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7 from peer 3b69c5d799e14e438df760453438aa75 at {username='slave'} at 127.0.99.134:33617: session id = 3b69c5d799e14e438df760453438aa75-3d8ae3c82c9d4c5c858fa3f9d5959eb7
I20240501 08:00:17.611807 7336 tablet_copy_source_session.cc:205] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.612818 10717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.613126 7317 consensus_queue.cc:237] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:17.615249 10717 tablet_copy_client.cc:783] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.615410 10717 tablet_copy_client.cc:647] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 1 WAL segments...
W20240501 08:00:17.616425 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b357c740aca144b7b063845726f419e2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.617687 7577 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.618211 9528 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.619657 7710 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.620146 9528 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.620950 10419 tablet_bootstrap.cc:492] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:17.621304 10419 tablet_bootstrap.cc:492] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap complete.
I20240501 08:00:17.621608 10419 ts_tablet_manager.cc:1367] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent bootstrapping tablet: real 0.261s user 0.022s sys 0.008s
I20240501 08:00:17.621618 10717 tablet_copy_client.cc:523] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.622310 10419 raft_consensus.cc:348] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:17.623620 10419 raft_consensus.cc:729] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8aa5ac376a54e1f8417e5af11d28fa7, State: Initialized, Role: LEARNER
I20240501 08:00:17.623818 10419 consensus_queue.cc:260] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:17.624425 10717 tablet_bootstrap.cc:492] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75: Bootstrap starting.
I20240501 08:00:17.624905 10419 ts_tablet_manager.cc:1398] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:00:17.625458 7467 tablet_copy_service.cc:342] P d6b4be7bbd1649d0b7c5e8efe81f759f: Request end of tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-3abb5202d8d24d9b9a67f7a87b7300a6 received from {username='slave'} at 127.0.99.133:43147
I20240501 08:00:17.625545 7467 tablet_copy_service.cc:433] P d6b4be7bbd1649d0b7c5e8efe81f759f: ending tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-3abb5202d8d24d9b9a67f7a87b7300a6 on tablet 3abb5202d8d24d9b9a67f7a87b7300a6 with peer d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:17.627142 9528 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:17.628341 10386 tablet_copy_client.cc:523] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.629783 10386 tablet_bootstrap.cc:492] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap starting.
I20240501 08:00:17.629774 7577 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:17.632450 7172 catalog_manager.cc:5484] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:17.637727 10728 ts_tablet_manager.cc:902] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:17.639683 10728 tablet_copy_client.cc:308] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:17.640138 10687 tablet_copy_client.cc:783] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.640383 10687 tablet_copy_client.cc:647] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.645100 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet bde42e485c4b49ce8acd694a25ba683a from peer 3b69c5d799e14e438df760453438aa75 ({username='slave'} at 127.0.99.134:51835)
I20240501 08:00:17.645212 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet bde42e485c4b49ce8acd694a25ba683a from peer 3b69c5d799e14e438df760453438aa75 at {username='slave'} at 127.0.99.134:51835: session id = 3b69c5d799e14e438df760453438aa75-bde42e485c4b49ce8acd694a25ba683a
I20240501 08:00:17.646167 7599 tablet_copy_source_session.cc:205] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.647058 10728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde42e485c4b49ce8acd694a25ba683a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.647204 10754 ts_tablet_manager.cc:902] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:17.647723 10754 tablet_copy_client.cc:308] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:00:17.648221 7730 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 4dde9710a5eb4baa881d9dd04affe184 from peer d8aa5ac376a54e1f8417e5af11d28fa7 ({username='slave'} at 127.0.99.133:35511)
I20240501 08:00:17.648344 7730 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 4dde9710a5eb4baa881d9dd04affe184 from peer d8aa5ac376a54e1f8417e5af11d28fa7 at {username='slave'} at 127.0.99.133:35511: session id = d8aa5ac376a54e1f8417e5af11d28fa7-4dde9710a5eb4baa881d9dd04affe184
I20240501 08:00:17.649260 7730 tablet_copy_source_session.cc:205] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.649571 10728 tablet_copy_client.cc:783] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.649854 10728 tablet_copy_client.cc:647] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.650230 10754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dde9710a5eb4baa881d9dd04affe184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.650789 7710 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:17.652431 10754 tablet_copy_client.cc:783] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.652592 10754 tablet_copy_client.cc:647] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.652627 10742 ts_tablet_manager.cc:902] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:17.653121 10742 tablet_copy_client.cc:308] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:00:17.654722 10728 tablet_copy_client.cc:523] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.656224 10728 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75: Bootstrap starting.
I20240501 08:00:17.659296 7730 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 81ccdd855b584d65860fd7a5777920a3 from peer 3b69c5d799e14e438df760453438aa75 ({username='slave'} at 127.0.99.134:52489)
I20240501 08:00:17.659408 7730 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 81ccdd855b584d65860fd7a5777920a3 from peer 3b69c5d799e14e438df760453438aa75 at {username='slave'} at 127.0.99.134:52489: session id = 3b69c5d799e14e438df760453438aa75-81ccdd855b584d65860fd7a5777920a3
I20240501 08:00:17.660295 7730 tablet_copy_source_session.cc:205] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.661442 10754 tablet_copy_client.cc:523] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.662277 10742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ccdd855b584d65860fd7a5777920a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.663138 10754 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap starting.
I20240501 08:00:17.664706 10742 tablet_copy_client.cc:783] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.664906 10742 tablet_copy_client.cc:647] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.667779 10770 ts_tablet_manager.cc:902] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:17.667774 10742 tablet_copy_client.cc:523] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.669656 10770 tablet_copy_client.cc:308] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:17.670181 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626 from peer 3b69c5d799e14e438df760453438aa75 ({username='slave'} at 127.0.99.134:33617)
I20240501 08:00:17.670308 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626 from peer 3b69c5d799e14e438df760453438aa75 at {username='slave'} at 127.0.99.134:33617: session id = 3b69c5d799e14e438df760453438aa75-b8fdc2b9c2ef44b0bfec59a1a2f6b626
I20240501 08:00:17.670434 10742 tablet_bootstrap.cc:492] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75: Bootstrap starting.
I20240501 08:00:17.671278 7336 tablet_copy_source_session.cc:205] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.674538 10770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.676970 10770 tablet_copy_client.cc:783] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.677208 10770 tablet_copy_client.cc:647] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.678304 10419 ts_tablet_manager.cc:902] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:17.679211 10419 tablet_copy_client.cc:308] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:17.681593 7317 consensus_queue.cc:237] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
W20240501 08:00:17.683265 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02b42b5fd3d440ea950b35ed461b8e12. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.683959 7710 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:17.684492 10264 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 08:00:17.683275 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02b42b5fd3d440ea950b35ed461b8e12. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.685204 10781 ts_tablet_manager.cc:902] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:17.685799 10781 tablet_copy_client.cc:308] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:17.685922 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet cc6f7dcc930e4937865d37cb3c3bb3d2 from peer d8aa5ac376a54e1f8417e5af11d28fa7 ({username='slave'} at 127.0.99.133:54767)
I20240501 08:00:17.686056 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet cc6f7dcc930e4937865d37cb3c3bb3d2 from peer d8aa5ac376a54e1f8417e5af11d28fa7 at {username='slave'} at 127.0.99.133:54767: session id = d8aa5ac376a54e1f8417e5af11d28fa7-cc6f7dcc930e4937865d37cb3c3bb3d2
I20240501 08:00:17.686489 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet 1088a116ccec4020aec562a69791a188 from peer 1c8cb9b1059a47cc822ca13a2b931776 ({username='slave'} at 127.0.99.135:39313)
I20240501 08:00:17.686594 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet 1088a116ccec4020aec562a69791a188 from peer 1c8cb9b1059a47cc822ca13a2b931776 at {username='slave'} at 127.0.99.135:39313: session id = 1c8cb9b1059a47cc822ca13a2b931776-1088a116ccec4020aec562a69791a188
I20240501 08:00:17.686614 10080 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.687108 7599 tablet_copy_source_session.cc:205] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.687438 7336 tablet_copy_source_session.cc:205] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.687767 7448 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:17.688391 10781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1088a116ccec4020aec562a69791a188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.688578 9528 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:17.689548 7448 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.689567 7710 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.691941 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc6f7dcc930e4937865d37cb3c3bb3d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.692281 7172 catalog_manager.cc:5484] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 4 to 5, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:17.694651 10419 tablet_copy_client.cc:783] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.694851 10419 tablet_copy_client.cc:647] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.700781 10687 tablet_copy_client.cc:523] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.703832 10687 tablet_bootstrap.cc:492] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap starting.
I20240501 08:00:17.705484 10386 tablet_bootstrap.cc:492] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:17.705794 10386 tablet_bootstrap.cc:492] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap complete.
I20240501 08:00:17.706084 10386 ts_tablet_manager.cc:1367] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent bootstrapping tablet: real 0.076s user 0.025s sys 0.003s
I20240501 08:00:17.706694 10386 raft_consensus.cc:348] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:17.706849 10386 raft_consensus.cc:729] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1807c0c2e7d04b20b0cda311c42eef8e, State: Initialized, Role: LEARNER
I20240501 08:00:17.706981 10386 consensus_queue.cc:260] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:17.707885 10386 ts_tablet_manager.cc:1398] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:17.708058 10419 tablet_copy_client.cc:523] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.708482 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-1e44704a9b9c46cf954d8e6937983841 received from {username='slave'} at 127.0.99.136:56099
I20240501 08:00:17.708588 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-1e44704a9b9c46cf954d8e6937983841 on tablet 1e44704a9b9c46cf954d8e6937983841 with peer 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:17.709371 10804 ts_tablet_manager.cc:902] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:17.709740 10419 tablet_bootstrap.cc:492] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap starting.
I20240501 08:00:17.713457 10770 tablet_copy_client.cc:523] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.714347 10804 tablet_copy_client.cc:308] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:17.714892 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet bde42e485c4b49ce8acd694a25ba683a from peer 1807c0c2e7d04b20b0cda311c42eef8e ({username='slave'} at 127.0.99.136:42359)
I20240501 08:00:17.715003 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet bde42e485c4b49ce8acd694a25ba683a from peer 1807c0c2e7d04b20b0cda311c42eef8e at {username='slave'} at 127.0.99.136:42359: session id = 1807c0c2e7d04b20b0cda311c42eef8e-bde42e485c4b49ce8acd694a25ba683a
I20240501 08:00:17.715160 10770 tablet_bootstrap.cc:492] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75: Bootstrap starting.
I20240501 08:00:17.716027 7599 tablet_copy_source_session.cc:205] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.717094 10804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde42e485c4b49ce8acd694a25ba683a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.720532 10804 tablet_copy_client.cc:783] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.720743 10804 tablet_copy_client.cc:647] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.721138 10781 tablet_copy_client.cc:783] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.721381 10781 tablet_copy_client.cc:647] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.723644 10386 ts_tablet_manager.cc:902] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:17.724084 10386 tablet_copy_client.cc:308] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:17.724516 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet 21d76484f5dd499fb6fc3d534bffb779 from peer 1807c0c2e7d04b20b0cda311c42eef8e ({username='slave'} at 127.0.99.136:42359)
I20240501 08:00:17.724627 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet 21d76484f5dd499fb6fc3d534bffb779 from peer 1807c0c2e7d04b20b0cda311c42eef8e at {username='slave'} at 127.0.99.136:42359: session id = 1807c0c2e7d04b20b0cda311c42eef8e-21d76484f5dd499fb6fc3d534bffb779
I20240501 08:00:17.725132 10781 tablet_copy_client.cc:523] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.725637 7599 tablet_copy_source_session.cc:205] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.726531 10386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d76484f5dd499fb6fc3d534bffb779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.728912 10386 tablet_copy_client.cc:783] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.729058 10386 tablet_copy_client.cc:647] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.732412 10499 tablet_bootstrap.cc:492] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:17.732767 10499 tablet_bootstrap.cc:492] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap complete.
I20240501 08:00:17.733099 10499 ts_tablet_manager.cc:1367] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent bootstrapping tablet: real 0.484s user 0.038s sys 0.001s
I20240501 08:00:17.733839 10386 tablet_copy_client.cc:523] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.735275 10386 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap starting.
I20240501 08:00:17.735786 10499 raft_consensus.cc:348] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:17.736058 10499 raft_consensus.cc:729] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c8cb9b1059a47cc822ca13a2b931776, State: Initialized, Role: LEARNER
I20240501 08:00:17.736443 10499 consensus_queue.cc:260] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:17.737427 10499 ts_tablet_manager.cc:1398] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 08:00:17.738328 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-5970772c02d04e0b95a2f2feb7dcd655 received from {username='slave'} at 127.0.99.135:39313
I20240501 08:00:17.738435 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-5970772c02d04e0b95a2f2feb7dcd655 on tablet 5970772c02d04e0b95a2f2feb7dcd655 with peer 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:17.739250 10615 tablet_bootstrap.cc:492] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:17.739517 10615 tablet_bootstrap.cc:492] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap complete.
I20240501 08:00:17.739671 10615 ts_tablet_manager.cc:1367] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent bootstrapping tablet: real 0.294s user 0.026s sys 0.003s
I20240501 08:00:17.740619 10615 raft_consensus.cc:348] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:17.740810 10615 raft_consensus.cc:729] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c8cb9b1059a47cc822ca13a2b931776, State: Initialized, Role: LEARNER
I20240501 08:00:17.741027 10615 consensus_queue.cc:260] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:17.741508 10615 ts_tablet_manager.cc:1398] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:17.742498 7467 tablet_copy_service.cc:342] P d6b4be7bbd1649d0b7c5e8efe81f759f: Request end of tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-3abb5202d8d24d9b9a67f7a87b7300a6 received from {username='slave'} at 127.0.99.135:41949
I20240501 08:00:17.742579 7467 tablet_copy_service.cc:433] P d6b4be7bbd1649d0b7c5e8efe81f759f: ending tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-3abb5202d8d24d9b9a67f7a87b7300a6 on tablet 3abb5202d8d24d9b9a67f7a87b7300a6 with peer 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:17.745030 10812 ts_tablet_manager.cc:902] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:17.746192 10812 tablet_copy_client.cc:308] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:17.746675 10817 ts_tablet_manager.cc:902] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:17.746905 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet 8311d0921e9e417090e441884615525a from peer d8aa5ac376a54e1f8417e5af11d28fa7 ({username='slave'} at 127.0.99.133:54767)
I20240501 08:00:17.747016 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet 8311d0921e9e417090e441884615525a from peer d8aa5ac376a54e1f8417e5af11d28fa7 at {username='slave'} at 127.0.99.133:54767: session id = d8aa5ac376a54e1f8417e5af11d28fa7-8311d0921e9e417090e441884615525a
I20240501 08:00:17.747282 10817 tablet_copy_client.cc:308] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:17.748060 7599 tablet_copy_source_session.cc:205] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.748443 10781 tablet_bootstrap.cc:492] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap starting.
I20240501 08:00:17.750859 10597 tablet_bootstrap.cc:492] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:17.751304 10597 tablet_bootstrap.cc:492] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75: Bootstrap complete.
I20240501 08:00:17.751667 10597 ts_tablet_manager.cc:1367] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75: Time spent bootstrapping tablet: real 0.351s user 0.037s sys 0.001s
I20240501 08:00:17.754344 10812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8311d0921e9e417090e441884615525a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.754894 10641 tablet_bootstrap.cc:492] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:17.755299 10641 tablet_bootstrap.cc:492] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap complete.
I20240501 08:00:17.759205 10804 tablet_copy_client.cc:523] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.761022 10804 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap starting.
I20240501 08:00:17.774472 10770 tablet_bootstrap.cc:492] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:17.762910 10831 ts_tablet_manager.cc:902] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:17.774837 10770 tablet_bootstrap.cc:492] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75: Bootstrap complete.
I20240501 08:00:17.775039 10770 ts_tablet_manager.cc:1367] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75: Time spent bootstrapping tablet: real 0.060s user 0.030s sys 0.000s
I20240501 08:00:17.755638 9056 raft_consensus.cc:1203] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 08:00:17.775405 10641 ts_tablet_manager.cc:1367] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent bootstrapping tablet: real 0.343s user 0.029s sys 0.000s
I20240501 08:00:17.776363 10831 tablet_copy_client.cc:308] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:17.776882 10641 raft_consensus.cc:348] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:17.777858 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet 21d76484f5dd499fb6fc3d534bffb779 from peer 3b69c5d799e14e438df760453438aa75 ({username='slave'} at 127.0.99.134:51835)
I20240501 08:00:17.777966 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet 21d76484f5dd499fb6fc3d534bffb779 from peer 3b69c5d799e14e438df760453438aa75 at {username='slave'} at 127.0.99.134:51835: session id = 3b69c5d799e14e438df760453438aa75-21d76484f5dd499fb6fc3d534bffb779
I20240501 08:00:17.778867 7599 tablet_copy_source_session.cc:205] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.780136 10831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d76484f5dd499fb6fc3d534bffb779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.780243 10641 raft_consensus.cc:729] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8aa5ac376a54e1f8417e5af11d28fa7, State: Initialized, Role: LEARNER
I20240501 08:00:17.781484 10641 consensus_queue.cc:260] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:17.782100 10641 ts_tablet_manager.cc:1398] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 08:00:17.787968 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-e178f7c3c17c4e1ca3d16b6441f76e77 received from {username='slave'} at 127.0.99.133:50977
I20240501 08:00:17.788089 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-e178f7c3c17c4e1ca3d16b6441f76e77 on tablet e178f7c3c17c4e1ca3d16b6441f76e77 with peer d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:17.788661 10615 ts_tablet_manager.cc:902] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:17.789222 10615 tablet_copy_client.cc:308] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:17.758049 10812 tablet_copy_client.cc:783] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.789799 10812 tablet_copy_client.cc:647] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.760648 10754 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:17.790299 10754 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap complete.
I20240501 08:00:17.790496 10754 ts_tablet_manager.cc:1367] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent bootstrapping tablet: real 0.127s user 0.029s sys 0.001s
I20240501 08:00:17.790638 10597 raft_consensus.cc:348] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:17.790983 10754 raft_consensus.cc:348] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:17.791150 10754 raft_consensus.cc:729] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8aa5ac376a54e1f8417e5af11d28fa7, State: Initialized, Role: LEARNER
I20240501 08:00:17.791280 10754 consensus_queue.cc:260] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:17.791342 10821 ts_tablet_manager.cc:902] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:17.791579 10754 ts_tablet_manager.cc:1398] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:17.790994 10831 tablet_copy_client.cc:783] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.791844 10831 tablet_copy_client.cc:647] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.792361 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-4dde9710a5eb4baa881d9dd04affe184 received from {username='slave'} at 127.0.99.133:35511
I20240501 08:00:17.792440 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-4dde9710a5eb4baa881d9dd04affe184 on tablet 4dde9710a5eb4baa881d9dd04affe184 with peer d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:17.793059 10821 tablet_copy_client.cc:308] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:17.793211 10770 raft_consensus.cc:348] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:17.793360 10770 raft_consensus.cc:729] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b69c5d799e14e438df760453438aa75, State: Initialized, Role: LEARNER
I20240501 08:00:17.793463 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet cc6f7dcc930e4937865d37cb3c3bb3d2 from peer 1807c0c2e7d04b20b0cda311c42eef8e ({username='slave'} at 127.0.99.136:42359)
I20240501 08:00:17.793558 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet cc6f7dcc930e4937865d37cb3c3bb3d2 from peer 1807c0c2e7d04b20b0cda311c42eef8e at {username='slave'} at 127.0.99.136:42359: session id = 1807c0c2e7d04b20b0cda311c42eef8e-cc6f7dcc930e4937865d37cb3c3bb3d2
I20240501 08:00:17.793749 10770 consensus_queue.cc:260] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:17.794507 7599 tablet_copy_source_session.cc:205] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.794723 10770 ts_tablet_manager.cc:1398] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240501 08:00:17.796427 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session 3b69c5d799e14e438df760453438aa75-b8fdc2b9c2ef44b0bfec59a1a2f6b626 received from {username='slave'} at 127.0.99.134:33617
I20240501 08:00:17.796505 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session 3b69c5d799e14e438df760453438aa75-b8fdc2b9c2ef44b0bfec59a1a2f6b626 on tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626 with peer 3b69c5d799e14e438df760453438aa75
I20240501 08:00:17.798954 10597 raft_consensus.cc:729] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b69c5d799e14e438df760453438aa75, State: Initialized, Role: LEARNER
I20240501 08:00:17.799146 10597 consensus_queue.cc:260] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:17.799629 10597 ts_tablet_manager.cc:1398] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75: Time spent starting tablet: real 0.027s user 0.004s sys 0.000s
I20240501 08:00:17.800068 7467 tablet_copy_service.cc:342] P d6b4be7bbd1649d0b7c5e8efe81f759f: Request end of tablet copy session 3b69c5d799e14e438df760453438aa75-f647cfc562794e65a05918b241014d72 received from {username='slave'} at 127.0.99.134:55743
I20240501 08:00:17.800164 7467 tablet_copy_service.cc:433] P d6b4be7bbd1649d0b7c5e8efe81f759f: ending tablet copy session 3b69c5d799e14e438df760453438aa75-f647cfc562794e65a05918b241014d72 on tablet f647cfc562794e65a05918b241014d72 with peer 3b69c5d799e14e438df760453438aa75
I20240501 08:00:17.800873 10831 tablet_copy_client.cc:523] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.802448 10831 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75: Bootstrap starting.
I20240501 08:00:17.810462 10821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc6f7dcc930e4937865d37cb3c3bb3d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.813891 10821 tablet_copy_client.cc:783] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.814286 10821 tablet_copy_client.cc:647] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.816051 10597 ts_tablet_manager.cc:902] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:17.816550 10597 tablet_copy_client.cc:308] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:00:17.816920 7598 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet bde42e485c4b49ce8acd694a25ba683a from peer 1c8cb9b1059a47cc822ca13a2b931776 ({username='slave'} at 127.0.99.135:36909)
I20240501 08:00:17.817034 7598 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet bde42e485c4b49ce8acd694a25ba683a from peer 1c8cb9b1059a47cc822ca13a2b931776 at {username='slave'} at 127.0.99.135:36909: session id = 1c8cb9b1059a47cc822ca13a2b931776-bde42e485c4b49ce8acd694a25ba683a
I20240501 08:00:17.817248 7728 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 5a37e631ae4f46a2be1ad5c43db5724b from peer 3b69c5d799e14e438df760453438aa75 ({username='slave'} at 127.0.99.134:52489)
I20240501 08:00:17.817364 7728 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 5a37e631ae4f46a2be1ad5c43db5724b from peer 3b69c5d799e14e438df760453438aa75 at {username='slave'} at 127.0.99.134:52489: session id = 3b69c5d799e14e438df760453438aa75-5a37e631ae4f46a2be1ad5c43db5724b
I20240501 08:00:17.817911 7598 tablet_copy_source_session.cc:205] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.818085 10821 tablet_copy_client.cc:523] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.816900 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet 21d76484f5dd499fb6fc3d534bffb779 from peer 1c8cb9b1059a47cc822ca13a2b931776 ({username='slave'} at 127.0.99.135:36909)
I20240501 08:00:17.818259 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet 21d76484f5dd499fb6fc3d534bffb779 from peer 1c8cb9b1059a47cc822ca13a2b931776 at {username='slave'} at 127.0.99.135:36909: session id = 1c8cb9b1059a47cc822ca13a2b931776-21d76484f5dd499fb6fc3d534bffb779
I20240501 08:00:17.818392 7728 tablet_copy_source_session.cc:205] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.819026 7599 tablet_copy_source_session.cc:205] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.819561 10615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde42e485c4b49ce8acd694a25ba683a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.820050 10817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d76484f5dd499fb6fc3d534bffb779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.821934 10817 tablet_copy_client.cc:783] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.822124 10817 tablet_copy_client.cc:647] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.822242 10615 tablet_copy_client.cc:783] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.822669 10597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a37e631ae4f46a2be1ad5c43db5724b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.823479 10817 tablet_copy_client.cc:523] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.824631 10817 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap starting.
I20240501 08:00:17.824995 10597 tablet_copy_client.cc:783] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.825160 10597 tablet_copy_client.cc:647] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.825248 10812 tablet_copy_client.cc:523] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.825670 10615 tablet_copy_client.cc:647] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.826849 10812 tablet_bootstrap.cc:492] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap starting.
I20240501 08:00:17.828097 10728 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:17.831036 10821 tablet_bootstrap.cc:492] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap starting.
I20240501 08:00:17.836591 10728 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75: Bootstrap complete.
I20240501 08:00:17.836866 10728 ts_tablet_manager.cc:1367] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75: Time spent bootstrapping tablet: real 0.181s user 0.024s sys 0.004s
I20240501 08:00:17.837469 10728 raft_consensus.cc:348] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:17.837693 10728 raft_consensus.cc:729] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b69c5d799e14e438df760453438aa75, State: Initialized, Role: LEARNER
I20240501 08:00:17.837863 10728 consensus_queue.cc:260] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:17.838373 10728 ts_tablet_manager.cc:1398] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:17.839933 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session 3b69c5d799e14e438df760453438aa75-bde42e485c4b49ce8acd694a25ba683a received from {username='slave'} at 127.0.99.134:51835
I20240501 08:00:17.840013 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session 3b69c5d799e14e438df760453438aa75-bde42e485c4b49ce8acd694a25ba683a on tablet bde42e485c4b49ce8acd694a25ba683a with peer 3b69c5d799e14e438df760453438aa75
I20240501 08:00:17.840310 10558 raft_consensus.cc:1050] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e to VOTER
I20240501 08:00:17.840847 10558 consensus_queue.cc:237] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:17.842614 7317 raft_consensus.cc:1261] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:17.843192 7448 raft_consensus.cc:1261] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:17.843631 10742 tablet_bootstrap.cc:492] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:17.843600 10559 consensus_queue.cc:1035] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:17.844069 10742 tablet_bootstrap.cc:492] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75: Bootstrap complete.
I20240501 08:00:17.844148 10559 consensus_queue.cc:1035] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:17.844385 10742 ts_tablet_manager.cc:1367] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75: Time spent bootstrapping tablet: real 0.174s user 0.027s sys 0.004s
I20240501 08:00:17.844846 10742 raft_consensus.cc:348] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:17.845029 10742 raft_consensus.cc:729] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b69c5d799e14e438df760453438aa75, State: Initialized, Role: LEARNER
I20240501 08:00:17.845180 10742 consensus_queue.cc:260] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:17.845631 10742 ts_tablet_manager.cc:1398] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:17.846144 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session 3b69c5d799e14e438df760453438aa75-81ccdd855b584d65860fd7a5777920a3 received from {username='slave'} at 127.0.99.134:52489
I20240501 08:00:17.846246 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session 3b69c5d799e14e438df760453438aa75-81ccdd855b584d65860fd7a5777920a3 on tablet 81ccdd855b584d65860fd7a5777920a3 with peer 3b69c5d799e14e438df760453438aa75
I20240501 08:00:17.847055 9449 raft_consensus.cc:1261] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:17.847262 10834 ts_tablet_manager.cc:902] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:17.848778 7577 consensus_queue.cc:237] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:17.849565 10533 consensus_queue.cc:1035] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:17.850697 7315 raft_consensus.cc:1261] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.851121 10533 consensus_queue.cc:1035] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:17.851881 7447 raft_consensus.cc:1261] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:17.853107 10559 consensus_queue.cc:1035] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:00:17.855888 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd07a00b6817400f86bf8c32bd592d8b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:17.856527 10834 tablet_copy_client.cc:308] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:00:17.857110 7728 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 4dde9710a5eb4baa881d9dd04affe184 from peer 1807c0c2e7d04b20b0cda311c42eef8e ({username='slave'} at 127.0.99.136:56099)
I20240501 08:00:17.857225 7728 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 4dde9710a5eb4baa881d9dd04affe184 from peer 1807c0c2e7d04b20b0cda311c42eef8e at {username='slave'} at 127.0.99.136:56099: session id = 1807c0c2e7d04b20b0cda311c42eef8e-4dde9710a5eb4baa881d9dd04affe184
I20240501 08:00:17.858175 7728 tablet_copy_source_session.cc:205] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.864666 10615 tablet_copy_client.cc:523] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.866319 10615 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap starting.
I20240501 08:00:17.872001 9319 raft_consensus.cc:1203] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 08:00:17.874379 10834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dde9710a5eb4baa881d9dd04affe184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.877095 10834 tablet_copy_client.cc:783] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.877331 10834 tablet_copy_client.cc:647] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.885262 10597 tablet_copy_client.cc:523] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.887065 10558 raft_consensus.cc:2932] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.889142 10558 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:17.889483 9449 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:17.891052 7447 raft_consensus.cc:2932] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.891606 7315 raft_consensus.cc:2932] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.894198 7169 catalog_manager.cc:5484] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:17.895108 7173 catalog_manager.cc:5484] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e reported cstate change: config changed from index 4 to 5, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:17.895608 9449 raft_consensus.cc:1203] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:00:17.897619 10597 tablet_bootstrap.cc:492] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75: Bootstrap starting.
I20240501 08:00:17.897711 7315 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:17.900211 7447 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:17.911567 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:17.912339 10499 ts_tablet_manager.cc:902] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:17.912845 10499 tablet_copy_client.cc:308] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:17.915763 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet 0e42ef8df4884bc9bc33f5ed4c164c8c from peer 1c8cb9b1059a47cc822ca13a2b931776 ({username='slave'} at 127.0.99.135:39313)
I20240501 08:00:17.915870 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet 0e42ef8df4884bc9bc33f5ed4c164c8c from peer 1c8cb9b1059a47cc822ca13a2b931776 at {username='slave'} at 127.0.99.135:39313: session id = 1c8cb9b1059a47cc822ca13a2b931776-0e42ef8df4884bc9bc33f5ed4c164c8c
I20240501 08:00:17.916265 7579 consensus_queue.cc:237] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:17.916798 7336 tablet_copy_source_session.cc:205] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.917672 9449 raft_consensus.cc:1261] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:17.917768 7447 raft_consensus.cc:1261] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:17.917837 10499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e42ef8df4884bc9bc33f5ed4c164c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.918161 10559 consensus_queue.cc:1035] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:17.919051 10559 consensus_queue.cc:1035] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:17.920209 10499 tablet_copy_client.cc:783] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:17.920362 10499 tablet_copy_client.cc:647] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:17.921202 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:17.922402 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:17.922435 9450 raft_consensus.cc:1203] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 08:00:17.922988 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:17.934213 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:17.937181 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.947993 10559 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:17.950071 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7377810e08104526b8b8acdf886a3368 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:17.950505 7447 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:17.951882 7175 catalog_manager.cc:5484] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 5 to 6, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:17.952922 9449 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:17.953119 10834 tablet_copy_client.cc:523] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.954401 10834 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap starting.
I20240501 08:00:17.956506 7297 tablet_service.cc:1514] Processing DeleteTablet for tablet 7377810e08104526b8b8acdf886a3368 with delete_type TABLET_DATA_TOMBSTONED (TS b4f8020266b5455b872a2973fc396578 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35594
I20240501 08:00:17.962219 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.969521 10386 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:17.969588 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:17.970165 10386 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap complete.
I20240501 08:00:17.971060 10386 ts_tablet_manager.cc:1367] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent bootstrapping tablet: real 0.236s user 0.023s sys 0.006s
I20240501 08:00:17.971798 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.973170 10386 raft_consensus.cc:348] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:17.973315 10386 raft_consensus.cc:729] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1807c0c2e7d04b20b0cda311c42eef8e, State: Initialized, Role: LEARNER
I20240501 08:00:17.973435 10386 consensus_queue.cc:260] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:17.973750 10386 ts_tablet_manager.cc:1398] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240501 08:00:17.974612 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-21d76484f5dd499fb6fc3d534bffb779 received from {username='slave'} at 127.0.99.136:42359
I20240501 08:00:17.974696 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-21d76484f5dd499fb6fc3d534bffb779 on tablet 21d76484f5dd499fb6fc3d534bffb779 with peer 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:17.974959 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:17.979910 10912 ts_tablet_manager.cc:902] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e: Initiating tablet copy from peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:17.980600 10912 tablet_copy_client.cc:308] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40707
I20240501 08:00:17.981169 7467 tablet_copy_service.cc:140] P d6b4be7bbd1649d0b7c5e8efe81f759f: Received BeginTabletCopySession request for tablet dd17760cd5c2454db97610b2f71c0ee2 from peer 1807c0c2e7d04b20b0cda311c42eef8e ({username='slave'} at 127.0.99.136:53469)
I20240501 08:00:17.981295 7467 tablet_copy_service.cc:161] P d6b4be7bbd1649d0b7c5e8efe81f759f: Beginning new tablet copy session on tablet dd17760cd5c2454db97610b2f71c0ee2 from peer 1807c0c2e7d04b20b0cda311c42eef8e at {username='slave'} at 127.0.99.136:53469: session id = 1807c0c2e7d04b20b0cda311c42eef8e-dd17760cd5c2454db97610b2f71c0ee2
I20240501 08:00:17.982326 7467 tablet_copy_source_session.cc:205] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:17.984463 10912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd17760cd5c2454db97610b2f71c0ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:17.987943 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.990679 10923 tablet_replica.cc:330] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578: stopping tablet replica
I20240501 08:00:17.990974 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:17.991112 10923 raft_consensus.cc:2229] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:17.991456 10923 raft_consensus.cc:2258] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:18.000684 10930 ts_tablet_manager.cc:902] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:18.011099 10923 ts_tablet_manager.cc:1875] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:18.011883 10638 tablet_bootstrap.cc:492] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.012250 10638 tablet_bootstrap.cc:492] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap complete.
I20240501 08:00:18.012441 10638 ts_tablet_manager.cc:1367] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent bootstrapping tablet: real 0.568s user 0.024s sys 0.004s
I20240501 08:00:18.012938 10638 raft_consensus.cc:348] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.013077 10638 raft_consensus.cc:729] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8aa5ac376a54e1f8417e5af11d28fa7, State: Initialized, Role: LEARNER
I20240501 08:00:18.015434 10912 tablet_copy_client.cc:783] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:18.015661 10912 tablet_copy_client.cc:647] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:18.020269 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:18.027218 10754 ts_tablet_manager.cc:902] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:18.027439 10912 tablet_copy_client.cc:523] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:18.027727 10754 tablet_copy_client.cc:308] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:18.028532 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet b357c740aca144b7b063845726f419e2 from peer d8aa5ac376a54e1f8417e5af11d28fa7 ({username='slave'} at 127.0.99.133:50977)
I20240501 08:00:18.028631 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet b357c740aca144b7b063845726f419e2 from peer d8aa5ac376a54e1f8417e5af11d28fa7 at {username='slave'} at 127.0.99.133:50977: session id = d8aa5ac376a54e1f8417e5af11d28fa7-b357c740aca144b7b063845726f419e2
I20240501 08:00:18.028929 10912 tablet_bootstrap.cc:492] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap starting.
I20240501 08:00:18.029485 7336 tablet_copy_source_session.cc:205] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:18.030603 10754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b357c740aca144b7b063845726f419e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:18.032788 10754 tablet_copy_client.cc:783] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:18.032941 10754 tablet_copy_client.cc:647] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:18.033244 9056 raft_consensus.cc:1203] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:00:18.035087 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:18.046051 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:18.051841 9319 raft_consensus.cc:1203] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:00:18.059490 10754 tablet_copy_client.cc:523] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:18.061666 10754 tablet_bootstrap.cc:492] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap starting.
I20240501 08:00:18.064389 10923 ts_tablet_manager.cc:1888] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:18.064570 10923 log.cc:1196] T 7377810e08104526b8b8acdf886a3368 P b4f8020266b5455b872a2973fc396578: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7377810e08104526b8b8acdf886a3368
I20240501 08:00:18.065253 7162 catalog_manager.cc:4830] TS b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): tablet 7377810e08104526b8b8acdf886a3368 (table rebalance_test_table_4 [id=ae84a8bf8ac745fb982412e9b194d7fc]) successfully deleted
I20240501 08:00:18.074966 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:18.081539 10638 consensus_queue.cc:260] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.109380 10499 tablet_copy_client.cc:523] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:18.111092 10499 tablet_bootstrap.cc:492] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap starting.
I20240501 08:00:18.114562 10638 ts_tablet_manager.cc:1398] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent starting tablet: real 0.102s user 0.001s sys 0.000s
I20240501 08:00:18.115109 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-1e44704a9b9c46cf954d8e6937983841 received from {username='slave'} at 127.0.99.133:35511
I20240501 08:00:18.115211 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-1e44704a9b9c46cf954d8e6937983841 on tablet 1e44704a9b9c46cf954d8e6937983841 with peer d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:18.120512 10930 tablet_copy_client.cc:308] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:00:18.121084 7728 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet df6dd3d8482d4931b1d3014a67e739c6 from peer 1c8cb9b1059a47cc822ca13a2b931776 ({username='slave'} at 127.0.99.135:43955)
I20240501 08:00:18.121192 7728 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet df6dd3d8482d4931b1d3014a67e739c6 from peer 1c8cb9b1059a47cc822ca13a2b931776 at {username='slave'} at 127.0.99.135:43955: session id = 1c8cb9b1059a47cc822ca13a2b931776-df6dd3d8482d4931b1d3014a67e739c6
I20240501 08:00:18.122124 7728 tablet_copy_source_session.cc:205] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:18.123103 10930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6dd3d8482d4931b1d3014a67e739c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:18.125494 10930 tablet_copy_client.cc:783] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:18.125732 10930 tablet_copy_client.cc:647] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:18.128926 10831 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.129149 7315 consensus_queue.cc:237] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:18.129585 10831 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75: Bootstrap complete.
I20240501 08:00:18.129806 10831 ts_tablet_manager.cc:1367] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75: Time spent bootstrapping tablet: real 0.327s user 0.023s sys 0.005s
I20240501 08:00:18.130378 10831 raft_consensus.cc:348] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
W20240501 08:00:18.131445 7251 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1088a116ccec4020aec562a69791a188. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.131974 7447 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.132241 7577 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 08:00:18.132448 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.027240 10419 tablet_bootstrap.cc:492] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.132602 10306 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.132869 10419 tablet_bootstrap.cc:492] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap complete.
I20240501 08:00:18.133076 10419 ts_tablet_manager.cc:1367] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent bootstrapping tablet: real 0.423s user 0.028s sys 0.000s
I20240501 08:00:18.133682 10419 raft_consensus.cc:348] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:18.133810 10419 raft_consensus.cc:729] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8aa5ac376a54e1f8417e5af11d28fa7, State: Initialized, Role: LEARNER
I20240501 08:00:18.133915 10419 consensus_queue.cc:260] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:18.134011 10080 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.135195 10419 ts_tablet_manager.cc:1398] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:18.135649 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-cc6f7dcc930e4937865d37cb3c3bb3d2 received from {username='slave'} at 127.0.99.133:54767
I20240501 08:00:18.135735 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-cc6f7dcc930e4937865d37cb3c3bb3d2 on tablet cc6f7dcc930e4937865d37cb3c3bb3d2 with peer d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:18.136359 9449 raft_consensus.cc:1203] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 08:00:18.150059 10930 tablet_copy_client.cc:523] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:18.151588 10930 tablet_bootstrap.cc:492] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap starting.
I20240501 08:00:18.154115 10817 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.154738 10817 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap complete.
I20240501 08:00:18.154954 10817 ts_tablet_manager.cc:1367] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent bootstrapping tablet: real 0.330s user 0.024s sys 0.005s
I20240501 08:00:18.155510 10817 raft_consensus.cc:348] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.155740 10817 raft_consensus.cc:729] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c8cb9b1059a47cc822ca13a2b931776, State: Initialized, Role: LEARNER
I20240501 08:00:18.155884 10817 consensus_queue.cc:260] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.156934 10817 ts_tablet_manager.cc:1398] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:18.157423 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-21d76484f5dd499fb6fc3d534bffb779 received from {username='slave'} at 127.0.99.135:36909
I20240501 08:00:18.157512 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-21d76484f5dd499fb6fc3d534bffb779 on tablet 21d76484f5dd499fb6fc3d534bffb779 with peer 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:18.180200 10080 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:18.183028 7448 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:18.185153 7577 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:18.187914 7710 consensus_queue.cc:237] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:18.188958 7172 catalog_manager.cc:5484] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 4 to 5, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:00:18.189745 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ea0db42a75a4879b3d3d608666dc2bf. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.190454 7448 raft_consensus.cc:1261] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:18.190743 7315 raft_consensus.cc:1261] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:18.190963 10323 consensus_queue.cc:1035] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:18.191143 8678 consensus_queue.cc:1035] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:18.195804 10821 tablet_bootstrap.cc:492] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.196128 10821 tablet_bootstrap.cc:492] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap complete.
I20240501 08:00:18.196324 10821 ts_tablet_manager.cc:1367] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent bootstrapping tablet: real 0.365s user 0.027s sys 0.000s
I20240501 08:00:18.196756 10687 tablet_bootstrap.cc:492] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.199031 10687 tablet_bootstrap.cc:492] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap complete.
I20240501 08:00:18.143611 10717 tablet_bootstrap.cc:492] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.199241 10831 raft_consensus.cc:729] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b69c5d799e14e438df760453438aa75, State: Initialized, Role: LEARNER
I20240501 08:00:18.199514 10717 tablet_bootstrap.cc:492] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75: Bootstrap complete.
I20240501 08:00:18.199697 10717 ts_tablet_manager.cc:1367] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75: Time spent bootstrapping tablet: real 0.575s user 0.028s sys 0.001s
I20240501 08:00:18.199393 10831 consensus_queue.cc:260] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.144762 9187 raft_consensus.cc:1203] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:00:18.200278 10831 ts_tablet_manager.cc:1398] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75: Time spent starting tablet: real 0.070s user 0.001s sys 0.000s
I20240501 08:00:18.179211 9188 raft_consensus.cc:1203] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:00:18.202484 10717 raft_consensus.cc:348] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:18.202610 10717 raft_consensus.cc:729] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b69c5d799e14e438df760453438aa75, State: Initialized, Role: LEARNER
I20240501 08:00:18.202713 10717 consensus_queue.cc:260] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:18.203023 10717 ts_tablet_manager.cc:1398] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:00:18.206568 8678 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:18.209461 7448 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:18.213371 10821 raft_consensus.cc:348] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:18.213541 10821 raft_consensus.cc:729] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1807c0c2e7d04b20b0cda311c42eef8e, State: Initialized, Role: LEARNER
I20240501 08:00:18.213654 10821 consensus_queue.cc:260] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:18.214217 10687 ts_tablet_manager.cc:1367] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent bootstrapping tablet: real 0.510s user 0.027s sys 0.003s
I20240501 08:00:18.214314 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session 3b69c5d799e14e438df760453438aa75-3d8ae3c82c9d4c5c858fa3f9d5959eb7 received from {username='slave'} at 127.0.99.134:33617
I20240501 08:00:18.214421 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session 3b69c5d799e14e438df760453438aa75-3d8ae3c82c9d4c5c858fa3f9d5959eb7 on tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7 with peer 3b69c5d799e14e438df760453438aa75
I20240501 08:00:18.214519 10821 ts_tablet_manager.cc:1398] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240501 08:00:18.215025 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-cc6f7dcc930e4937865d37cb3c3bb3d2 received from {username='slave'} at 127.0.99.136:42359
I20240501 08:00:18.215106 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-cc6f7dcc930e4937865d37cb3c3bb3d2 on tablet cc6f7dcc930e4937865d37cb3c3bb3d2 with peer 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:18.203480 10615 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.216594 10742 ts_tablet_manager.cc:902] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:18.216905 10615 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap complete.
I20240501 08:00:18.217042 10742 tablet_copy_client.cc:308] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:18.217089 10615 ts_tablet_manager.cc:1367] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776: Time spent bootstrapping tablet: real 0.351s user 0.029s sys 0.000s
I20240501 08:00:18.217542 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet 02b42b5fd3d440ea950b35ed461b8e12 from peer 3b69c5d799e14e438df760453438aa75 ({username='slave'} at 127.0.99.134:33617)
I20240501 08:00:18.217630 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet 02b42b5fd3d440ea950b35ed461b8e12 from peer 3b69c5d799e14e438df760453438aa75 at {username='slave'} at 127.0.99.134:33617: session id = 3b69c5d799e14e438df760453438aa75-02b42b5fd3d440ea950b35ed461b8e12
I20240501 08:00:18.217957 7175 catalog_manager.cc:5484] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:18.218530 7336 tablet_copy_source_session.cc:205] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:18.220489 9188 raft_consensus.cc:1203] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 08:00:18.221413 10742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b42b5fd3d440ea950b35ed461b8e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:18.223385 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session 3b69c5d799e14e438df760453438aa75-21d76484f5dd499fb6fc3d534bffb779 received from {username='slave'} at 127.0.99.134:51835
I20240501 08:00:18.223474 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session 3b69c5d799e14e438df760453438aa75-21d76484f5dd499fb6fc3d534bffb779 on tablet 21d76484f5dd499fb6fc3d534bffb779 with peer 3b69c5d799e14e438df760453438aa75
I20240501 08:00:18.224220 10742 tablet_copy_client.cc:783] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:18.224350 10742 tablet_copy_client.cc:647] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:18.226625 9449 raft_consensus.cc:1203] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:00:18.227685 10812 tablet_bootstrap.cc:492] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.228065 10812 tablet_bootstrap.cc:492] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap complete.
I20240501 08:00:18.228922 9188 raft_consensus.cc:1203] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:00:18.230234 10812 ts_tablet_manager.cc:1367] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent bootstrapping tablet: real 0.403s user 0.025s sys 0.003s
I20240501 08:00:18.230752 10812 raft_consensus.cc:348] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:18.230949 10812 raft_consensus.cc:729] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8aa5ac376a54e1f8417e5af11d28fa7, State: Initialized, Role: LEARNER
I20240501 08:00:18.231146 10812 consensus_queue.cc:260] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:18.231278 10615 raft_consensus.cc:348] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.231467 10615 raft_consensus.cc:729] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c8cb9b1059a47cc822ca13a2b931776, State: Initialized, Role: LEARNER
I20240501 08:00:18.231618 10615 consensus_queue.cc:260] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.232473 10804 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.232796 10804 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap complete.
I20240501 08:00:18.233001 10804 ts_tablet_manager.cc:1367] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent bootstrapping tablet: real 0.472s user 0.029s sys 0.000s
I20240501 08:00:18.233489 10804 raft_consensus.cc:348] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.233624 10804 raft_consensus.cc:729] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1807c0c2e7d04b20b0cda311c42eef8e, State: Initialized, Role: LEARNER
I20240501 08:00:18.233729 10804 consensus_queue.cc:260] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.233938 10615 ts_tablet_manager.cc:1398] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20240501 08:00:18.234335 10804 ts_tablet_manager.cc:1398] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 08:00:18.234772 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-bde42e485c4b49ce8acd694a25ba683a received from {username='slave'} at 127.0.99.136:42359
I20240501 08:00:18.234851 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-bde42e485c4b49ce8acd694a25ba683a on tablet bde42e485c4b49ce8acd694a25ba683a with peer 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:18.235369 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-bde42e485c4b49ce8acd694a25ba683a received from {username='slave'} at 127.0.99.135:36909
I20240501 08:00:18.235440 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-bde42e485c4b49ce8acd694a25ba683a on tablet bde42e485c4b49ce8acd694a25ba683a with peer 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:18.236680 10687 raft_consensus.cc:348] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.236858 10687 raft_consensus.cc:729] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c8cb9b1059a47cc822ca13a2b931776, State: Initialized, Role: LEARNER
I20240501 08:00:18.236994 10687 consensus_queue.cc:260] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.239185 7315 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:18.239753 10597 tablet_bootstrap.cc:492] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.239746 10687 ts_tablet_manager.cc:1398] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent starting tablet: real 0.025s user 0.001s sys 0.000s
I20240501 08:00:18.241715 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-81ccdd855b584d65860fd7a5777920a3 received from {username='slave'} at 127.0.99.135:43955
I20240501 08:00:18.241799 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-81ccdd855b584d65860fd7a5777920a3 on tablet 81ccdd855b584d65860fd7a5777920a3 with peer 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:18.241439 10419 ts_tablet_manager.cc:902] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:18.254266 10419 tablet_copy_client.cc:308] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:18.254699 10812 ts_tablet_manager.cc:1398] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent starting tablet: real 0.024s user 0.001s sys 0.000s
I20240501 08:00:18.255303 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet 02b42b5fd3d440ea950b35ed461b8e12 from peer d8aa5ac376a54e1f8417e5af11d28fa7 ({username='slave'} at 127.0.99.133:50977)
I20240501 08:00:18.255453 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet 02b42b5fd3d440ea950b35ed461b8e12 from peer d8aa5ac376a54e1f8417e5af11d28fa7 at {username='slave'} at 127.0.99.133:50977: session id = d8aa5ac376a54e1f8417e5af11d28fa7-02b42b5fd3d440ea950b35ed461b8e12
I20240501 08:00:18.256319 7336 tablet_copy_source_session.cc:205] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:18.253522 10742 tablet_copy_client.cc:523] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:18.257078 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-8311d0921e9e417090e441884615525a received from {username='slave'} at 127.0.99.133:54767
I20240501 08:00:18.257156 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-8311d0921e9e417090e441884615525a on tablet 8311d0921e9e417090e441884615525a with peer d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:18.258042 10742 tablet_bootstrap.cc:492] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75: Bootstrap starting.
I20240501 08:00:18.258234 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b42b5fd3d440ea950b35ed461b8e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:18.260483 10419 tablet_copy_client.cc:783] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:18.260632 10419 tablet_copy_client.cc:647] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:18.240072 10597 tablet_bootstrap.cc:492] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75: Bootstrap complete.
I20240501 08:00:18.261265 10597 ts_tablet_manager.cc:1367] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75: Time spent bootstrapping tablet: real 0.364s user 0.029s sys 0.000s
I20240501 08:00:18.262234 10834 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.262599 10834 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap complete.
I20240501 08:00:18.262825 10834 ts_tablet_manager.cc:1367] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent bootstrapping tablet: real 0.308s user 0.029s sys 0.000s
I20240501 08:00:18.262984 10597 raft_consensus.cc:348] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:18.263115 10597 raft_consensus.cc:729] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b69c5d799e14e438df760453438aa75, State: Initialized, Role: LEARNER
I20240501 08:00:18.263235 10597 consensus_queue.cc:260] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:18.263504 10834 raft_consensus.cc:348] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.265275 7315 consensus_queue.cc:237] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:18.265661 10781 tablet_bootstrap.cc:492] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.265810 10834 raft_consensus.cc:729] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1807c0c2e7d04b20b0cda311c42eef8e, State: Initialized, Role: LEARNER
I20240501 08:00:18.266212 10834 consensus_queue.cc:260] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.266702 10597 ts_tablet_manager.cc:1398] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 08:00:18.267146 10834 ts_tablet_manager.cc:1398] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:00:18.267519 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session 3b69c5d799e14e438df760453438aa75-5a37e631ae4f46a2be1ad5c43db5724b received from {username='slave'} at 127.0.99.134:52489
I20240501 08:00:18.267601 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session 3b69c5d799e14e438df760453438aa75-5a37e631ae4f46a2be1ad5c43db5724b on tablet 5a37e631ae4f46a2be1ad5c43db5724b with peer 3b69c5d799e14e438df760453438aa75
I20240501 08:00:18.268018 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-4dde9710a5eb4baa881d9dd04affe184 received from {username='slave'} at 127.0.99.136:56099
I20240501 08:00:18.268087 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-4dde9710a5eb4baa881d9dd04affe184 on tablet 4dde9710a5eb4baa881d9dd04affe184 with peer 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:18.268455 9188 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 08:00:18.269264 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d8ae3c82c9d4c5c858fa3f9d5959eb7. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.269277 10264 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.269970 7448 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.270342 7710 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.270713 10264 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.272117 10264 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.272979 10912 tablet_bootstrap.cc:492] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.273291 10912 tablet_bootstrap.cc:492] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap complete.
I20240501 08:00:18.273751 10419 tablet_copy_client.cc:523] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:18.275313 10419 tablet_bootstrap.cc:492] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap starting.
I20240501 08:00:18.275430 10912 ts_tablet_manager.cc:1367] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent bootstrapping tablet: real 0.247s user 0.019s sys 0.010s
I20240501 08:00:18.275974 10912 raft_consensus.cc:348] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.276085 10912 raft_consensus.cc:729] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1807c0c2e7d04b20b0cda311c42eef8e, State: Initialized, Role: LEARNER
I20240501 08:00:18.276180 10912 consensus_queue.cc:260] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.277226 10912 ts_tablet_manager.cc:1398] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:18.278654 7467 tablet_copy_service.cc:342] P d6b4be7bbd1649d0b7c5e8efe81f759f: Request end of tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-dd17760cd5c2454db97610b2f71c0ee2 received from {username='slave'} at 127.0.99.136:53469
I20240501 08:00:18.278733 7467 tablet_copy_service.cc:433] P d6b4be7bbd1649d0b7c5e8efe81f759f: ending tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-dd17760cd5c2454db97610b2f71c0ee2 on tablet dd17760cd5c2454db97610b2f71c0ee2 with peer 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:18.291656 10754 tablet_bootstrap.cc:492] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.296721 10974 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:18.300642 7448 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:18.303632 7174 catalog_manager.cc:5484] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 4 to 5, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:18.305619 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:18.306286 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:18.308876 7448 consensus_queue.cc:237] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.310745 7708 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:18.310760 7315 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:18.311272 10381 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:18.311594 10381 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:00:18.311713 7382 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffc23756b44d4d588e0a8175075d9bc6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.315131 9056 raft_consensus.cc:1203] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:00:18.318368 10419 tablet_bootstrap.cc:492] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.319648 10742 tablet_bootstrap.cc:492] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.319941 10742 tablet_bootstrap.cc:492] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75: Bootstrap complete.
I20240501 08:00:18.320115 10742 ts_tablet_manager.cc:1367] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75: Time spent bootstrapping tablet: real 0.062s user 0.027s sys 0.000s
I20240501 08:00:18.320598 10742 raft_consensus.cc:348] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:18.320731 10742 raft_consensus.cc:729] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b69c5d799e14e438df760453438aa75, State: Initialized, Role: LEARNER
I20240501 08:00:18.320837 10742 consensus_queue.cc:260] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:18.321887 10147 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.324354 10742 ts_tablet_manager.cc:1398] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:00:18.324551 7315 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.328171 7169 catalog_manager.cc:5484] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.328908 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session 3b69c5d799e14e438df760453438aa75-02b42b5fd3d440ea950b35ed461b8e12 received from {username='slave'} at 127.0.99.134:33617
I20240501 08:00:18.328999 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session 3b69c5d799e14e438df760453438aa75-02b42b5fd3d440ea950b35ed461b8e12 on tablet 02b42b5fd3d440ea950b35ed461b8e12 with peer 3b69c5d799e14e438df760453438aa75
I20240501 08:00:18.329867 7708 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.265969 10781 tablet_bootstrap.cc:492] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap complete.
I20240501 08:00:18.338367 7708 consensus_queue.cc:237] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:18.338598 9528 raft_consensus.cc:1050] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER 3b69c5d799e14e438df760453438aa75 to VOTER
I20240501 08:00:18.287334 10930 tablet_bootstrap.cc:492] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.338860 10781 ts_tablet_manager.cc:1367] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent bootstrapping tablet: real 0.590s user 0.019s sys 0.011s
I20240501 08:00:18.339040 10930 tablet_bootstrap.cc:492] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap complete.
I20240501 08:00:18.339100 9528 consensus_queue.cc:237] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:18.339218 10930 ts_tablet_manager.cc:1367] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent bootstrapping tablet: real 0.188s user 0.022s sys 0.007s
I20240501 08:00:18.339404 10781 raft_consensus.cc:348] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.339524 10781 raft_consensus.cc:729] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c8cb9b1059a47cc822ca13a2b931776, State: Initialized, Role: LEARNER
I20240501 08:00:18.339623 10781 consensus_queue.cc:260] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.339660 10930 raft_consensus.cc:348] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.339784 10930 raft_consensus.cc:729] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c8cb9b1059a47cc822ca13a2b931776, State: Initialized, Role: LEARNER
I20240501 08:00:18.339866 10930 consensus_queue.cc:260] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.340111 10930 ts_tablet_manager.cc:1398] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:18.340688 10781 ts_tablet_manager.cc:1398] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:18.342994 10419 tablet_bootstrap.cc:492] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap complete.
I20240501 08:00:18.343127 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-df6dd3d8482d4931b1d3014a67e739c6 received from {username='slave'} at 127.0.99.135:43955
I20240501 08:00:18.343222 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-df6dd3d8482d4931b1d3014a67e739c6 on tablet df6dd3d8482d4931b1d3014a67e739c6 with peer 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:18.343210 10419 ts_tablet_manager.cc:1367] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent bootstrapping tablet: real 0.068s user 0.029s sys 0.001s
I20240501 08:00:18.343701 10419 raft_consensus.cc:348] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:18.343825 10419 raft_consensus.cc:729] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8aa5ac376a54e1f8417e5af11d28fa7, State: Initialized, Role: LEARNER
I20240501 08:00:18.343845 7710 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.343876 7448 raft_consensus.cc:1261] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:18.343930 10419 consensus_queue.cc:260] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:18.344215 10323 consensus_queue.cc:1035] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:18.344370 10419 ts_tablet_manager.cc:1398] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:18.344776 10754 tablet_bootstrap.cc:492] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap complete.
I20240501 08:00:18.344940 10754 ts_tablet_manager.cc:1367] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent bootstrapping tablet: real 0.283s user 0.028s sys 0.000s
I20240501 08:00:18.345299 10754 raft_consensus.cc:348] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:18.345412 10754 raft_consensus.cc:729] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8aa5ac376a54e1f8417e5af11d28fa7, State: Initialized, Role: LEARNER
I20240501 08:00:18.345494 10754 consensus_queue.cc:260] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:18.345129 9187 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:18.345731 10754 ts_tablet_manager.cc:1398] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:18.347074 9186 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.348683 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-1088a116ccec4020aec562a69791a188 received from {username='slave'} at 127.0.99.135:39313
I20240501 08:00:18.348770 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-1088a116ccec4020aec562a69791a188 on tablet 1088a116ccec4020aec562a69791a188 with peer 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:18.349960 7337 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-b357c740aca144b7b063845726f419e2 received from {username='slave'} at 127.0.99.133:50977
I20240501 08:00:18.350080 7337 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-b357c740aca144b7b063845726f419e2 on tablet b357c740aca144b7b063845726f419e2 with peer d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:18.350215 10147 raft_consensus.cc:1050] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f: attempting to promote NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 to VOTER
I20240501 08:00:18.350275 10306 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.350673 10147 consensus_queue.cc:237] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
W20240501 08:00:18.351312 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d8ae3c82c9d4c5c858fa3f9d5959eb7. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:18.351965 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07967f68a412411494c6dd1f47209009. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.352162 9449 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.352247 10080 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.352559 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-02b42b5fd3d440ea950b35ed461b8e12 received from {username='slave'} at 127.0.99.133:50977
I20240501 08:00:18.352607 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-02b42b5fd3d440ea950b35ed461b8e12 on tablet 02b42b5fd3d440ea950b35ed461b8e12 with peer d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:18.352728 10383 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.353540 9186 raft_consensus.cc:1203] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 08:00:18.354431 7710 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:18.355607 7448 consensus_queue.cc:237] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.357167 7315 raft_consensus.cc:1261] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:18.357321 7577 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:18.358119 7710 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:18.358515 10381 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:00:18.359263 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d18a2918c75467a9fe59979ce8936cd. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.359371 11045 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:18.359503 10323 consensus_queue.cc:1035] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:18.360430 7448 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.360908 10080 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.362056 9056 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.362497 10147 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.362555 9318 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.362915 7579 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.363564 10383 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.363670 11049 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.364192 7317 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.364591 10381 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.367362 10080 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:18.367918 9186 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:18.368526 7448 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:18.370165 9056 raft_consensus.cc:1203] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:00:18.370615 11045 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.371883 9317 raft_consensus.cc:1203] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 08:00:18.372531 7579 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.373927 7168 catalog_manager.cc:5484] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 5 to 6, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:18.375142 7708 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:18.376358 9056 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.378690 7317 consensus_queue.cc:237] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:18.378630 9318 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.381213 7448 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.381793 10974 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.381737 7168 catalog_manager.cc:5484] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
W20240501 08:00:18.383226 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d8ae3c82c9d4c5c858fa3f9d5959eb7. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.383358 9186 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.383399 7708 consensus_queue.cc:237] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.383983 11045 raft_consensus.cc:1050] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f: attempting to promote NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 to VOTER
I20240501 08:00:18.384373 11045 consensus_queue.cc:237] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:18.384930 10974 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.385221 10974 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
W20240501 08:00:18.385434 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81ccdd855b584d65860fd7a5777920a3. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.385619 9318 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.385967 10323 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.386054 10383 raft_consensus.cc:1013] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: attempt to promote peer 1c8cb9b1059a47cc822ca13a2b931776: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:00:18.386198 7579 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.386687 10323 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.387140 7448 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.387528 10323 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.388217 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.388828 9187 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.389127 10323 raft_consensus.cc:1013] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: attempt to promote peer 1c8cb9b1059a47cc822ca13a2b931776: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:00:18.390038 7169 catalog_manager.cc:5484] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 6 to 7, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:18.390378 7447 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:18.391237 9186 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:18.389911 7317 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.393716 7688 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7 with delete_type TABLET_DATA_TOMBSTONED (TS 09a7b6e2731a474b80411efb62323604 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49464
I20240501 08:00:18.394145 10335 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.394218 9318 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.394910 10383 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.397038 9056 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.398059 7577 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.398459 7317 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.398835 10383 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.398941 10381 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.399165 10383 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.401021 10912 ts_tablet_manager.cc:902] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:18.401458 10912 tablet_copy_client.cc:308] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:18.401487 10323 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:18.401707 11045 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.403496 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet cd07a00b6817400f86bf8c32bd592d8b from peer 1807c0c2e7d04b20b0cda311c42eef8e ({username='slave'} at 127.0.99.136:42359)
I20240501 08:00:18.403587 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet cd07a00b6817400f86bf8c32bd592d8b from peer 1807c0c2e7d04b20b0cda311c42eef8e at {username='slave'} at 127.0.99.136:42359: session id = 1807c0c2e7d04b20b0cda311c42eef8e-cd07a00b6817400f86bf8c32bd592d8b
I20240501 08:00:18.404577 7599 tablet_copy_source_session.cc:205] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:18.405540 10912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd07a00b6817400f86bf8c32bd592d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:18.405450 7710 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.406792 7446 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:18.407924 10912 tablet_copy_client.cc:783] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:18.408074 10912 tablet_copy_client.cc:647] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:18.408725 9450 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.408808 7174 catalog_manager.cc:5484] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:18.408898 11045 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.409574 10383 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.409644 7578 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.410926 9450 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.411134 11058 tablet_replica.cc:330] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604: stopping tablet replica
I20240501 08:00:18.411484 11058 raft_consensus.cc:2229] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:18.411698 11058 raft_consensus.cc:2258] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:18.412446 9450 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.412534 7168 catalog_manager.cc:5484] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index -1 to 4, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.412688 10335 raft_consensus.cc:1013] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: attempt to promote peer 3b69c5d799e14e438df760453438aa75: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:00:18.413957 11058 ts_tablet_manager.cc:1875] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:00:18.414943 7159 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 3abb5202d8d24d9b9a67f7a87b7300a6 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 7 but the committed config has opid_index of 8
I20240501 08:00:18.415004 9318 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.415516 7175 catalog_manager.cc:5484] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e reported cstate change: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.415750 7315 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:18.417253 7710 consensus_queue.cc:237] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
W20240501 08:00:18.419538 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ea0db42a75a4879b3d3d608666dc2bf. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:18.419648 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ea0db42a75a4879b3d3d608666dc2bf. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.331277 9319 raft_consensus.cc:1203] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 08:00:18.420264 11058 ts_tablet_manager.cc:1888] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:00:18.420346 11058 log.cc:1196] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 09a7b6e2731a474b80411efb62323604: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3d8ae3c82c9d4c5c858fa3f9d5959eb7
I20240501 08:00:18.420419 7315 raft_consensus.cc:1261] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.420789 8678 consensus_queue.cc:1035] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.421197 7162 catalog_manager.cc:4830] TS 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789): tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7 (table rebalance_test_table_2 [id=d578050a40e0474484896c98e4da9f5a]) successfully deleted
I20240501 08:00:18.422010 9056 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.422667 7447 raft_consensus.cc:1261] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.424022 8678 consensus_queue.cc:1035] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.425949 8678 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.427904 7447 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.428056 7315 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.431310 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:18.431406 10147 raft_consensus.cc:1050] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f: attempting to promote NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e to VOTER
I20240501 08:00:18.431308 7577 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.431816 10147 consensus_queue.cc:237] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:18.434388 7577 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.434758 11049 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:18.434983 9319 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.435813 9187 raft_consensus.cc:1203] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:00:18.436631 7172 catalog_manager.cc:5484] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:18.438305 9449 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.438694 11049 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:18.439258 11049 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:00:18.440747 7159 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 3abb5202d8d24d9b9a67f7a87b7300a6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER failed with leader d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:18.441716 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:18.442662 9056 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.443056 11049 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:18.447871 11049 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.450469 9319 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.451336 7577 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.452140 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:18.452934 9056 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.454684 7172 catalog_manager.cc:5484] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 8 to 9, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.457489 9450 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.460732 7315 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.461578 11045 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.465440 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:18.466289 7315 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.468349 7447 consensus_queue.cc:237] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:18.295609 10499 tablet_bootstrap.cc:492] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.469051 10499 tablet_bootstrap.cc:492] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap complete.
I20240501 08:00:18.469265 10499 ts_tablet_manager.cc:1367] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776: Time spent bootstrapping tablet: real 0.358s user 0.027s sys 0.000s
I20240501 08:00:18.469738 7315 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.469828 10499 raft_consensus.cc:348] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.469945 10499 raft_consensus.cc:729] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c8cb9b1059a47cc822ca13a2b931776, State: Initialized, Role: LEARNER
I20240501 08:00:18.470072 10499 consensus_queue.cc:260] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.470479 10499 ts_tablet_manager.cc:1398] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:18.470634 9319 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:18.471400 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-0e42ef8df4884bc9bc33f5ed4c164c8c received from {username='slave'} at 127.0.99.135:39313
I20240501 08:00:18.471477 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-0e42ef8df4884bc9bc33f5ed4c164c8c on tablet 0e42ef8df4884bc9bc33f5ed4c164c8c with peer 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:18.471951 11049 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:18.473666 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:18.475628 7577 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:18.476284 9450 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:18.476804 10147 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:18.477102 10147 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:18.485122 9056 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:18.485708 11049 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:18.486578 11049 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.487735 9449 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.490029 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3abb5202d8d24d9b9a67f7a87b7300a6 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.490140 9319 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.490278 7168 catalog_manager.cc:5484] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e reported cstate change: config changed from index 9 to 10, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.491135 7577 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.493391 7159 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 3abb5202d8d24d9b9a67f7a87b7300a6 with cas_config_opid_index 8: aborting the task: latest config opid_index 10; task opid_index 8
I20240501 08:00:18.494259 7297 tablet_service.cc:1514] Processing DeleteTablet for tablet 3abb5202d8d24d9b9a67f7a87b7300a6 with delete_type TABLET_DATA_TOMBSTONED (TS b4f8020266b5455b872a2973fc396578 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35594
I20240501 08:00:18.494525 10923 tablet_replica.cc:330] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578: stopping tablet replica
I20240501 08:00:18.494695 10923 raft_consensus.cc:2229] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:18.494937 10923 raft_consensus.cc:2258] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:18.495991 9056 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.499078 10923 ts_tablet_manager.cc:1875] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:18.500718 7448 consensus_queue.cc:237] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:18.501190 10923 ts_tablet_manager.cc:1888] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 08:00:18.501255 10923 log.cc:1196] T 3abb5202d8d24d9b9a67f7a87b7300a6 P b4f8020266b5455b872a2973fc396578: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/3abb5202d8d24d9b9a67f7a87b7300a6
I20240501 08:00:18.501899 7162 catalog_manager.cc:4830] TS b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): tablet 3abb5202d8d24d9b9a67f7a87b7300a6 (table rebalance_test_table_0 [id=61deeb642b0b4f06a2ef4b1a391fdcb9]) successfully deleted
I20240501 08:00:18.503099 9449 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:18.503494 10147 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:18.503751 9319 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:18.504139 10147 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:18.505041 10323 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.506291 9056 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:18.506747 10383 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:18.507807 9187 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.508597 10912 tablet_copy_client.cc:523] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:18.508791 11074 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.509573 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:18.510159 10912 tablet_bootstrap.cc:492] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap starting.
I20240501 08:00:18.510403 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3abb5202d8d24d9b9a67f7a87b7300a6 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.510990 11073 raft_consensus.cc:1050] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER 3b69c5d799e14e438df760453438aa75 to VOTER
I20240501 08:00:18.511581 7447 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.511710 7579 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.511902 9318 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.513329 9057 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.515087 7169 catalog_manager.cc:5484] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 reported cstate change: config changed from index 10 to 11, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.515048 7174 catalog_manager.cc:5484] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.515053 9319 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.515791 11073 consensus_queue.cc:237] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.517788 8678 raft_consensus.cc:1050] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e to VOTER
I20240501 08:00:18.518254 8678 consensus_queue.cc:237] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.518550 7558 tablet_service.cc:1514] Processing DeleteTablet for tablet 3abb5202d8d24d9b9a67f7a87b7300a6 with delete_type TABLET_DATA_TOMBSTONED (TS 31b02f06be8a4455b94b1254d77005bd not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:33196
I20240501 08:00:18.519960 9057 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.520388 11072 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.520500 7315 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.520905 11072 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.522046 9448 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.524639 7448 consensus_queue.cc:237] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
W20240501 08:00:18.524842 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81ccdd855b584d65860fd7a5777920a3. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.525610 7579 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.526111 10335 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.526206 9318 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.526448 10335 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.528681 9318 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:00:18.529086 11075 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:00:18.531927 11072 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.532326 9187 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.532720 11072 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.533651 9056 raft_consensus.cc:1261] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:00:18.533841 9318 raft_consensus.cc:1203] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:00:18.534123 7448 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.534348 9319 raft_consensus.cc:1203] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:00:18.535465 11072 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.535789 11072 raft_consensus.cc:1013] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: attempt to promote peer 1c8cb9b1059a47cc822ca13a2b931776: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:00:18.535799 11074 consensus_queue.cc:1035] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:00:18.535997 11045 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.536471 9319 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.538204 9450 raft_consensus.cc:1203] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:00:18.538208 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3abb5202d8d24d9b9a67f7a87b7300a6 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.539521 7169 catalog_manager.cc:5484] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1c8cb9b1059a47cc822ca13a2b931776 reported cstate change: config changed from index 11 to 12, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.539660 9056 raft_consensus.cc:2932] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.545769 7577 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.546217 11073 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.546633 9430 tablet_service.cc:1514] Processing DeleteTablet for tablet 3abb5202d8d24d9b9a67f7a87b7300a6 with delete_type TABLET_DATA_TOMBSTONED (TS 1807c0c2e7d04b20b0cda311c42eef8e not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:58736
I20240501 08:00:18.548605 10080 raft_consensus.cc:1050] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 to VOTER
I20240501 08:00:18.548991 10080 consensus_queue.cc:237] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:18.550451 9319 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.550830 11040 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.552934 7448 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.553303 11040 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.554471 7710 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.555263 11040 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.557415 7317 consensus_queue.cc:237] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.558418 10323 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.559937 9056 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 08:00:18.559981 7250 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e178f7c3c17c4e1ca3d16b6441f76e77. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.560286 7578 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.560635 7447 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.560649 10080 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.560813 11040 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.561051 11040 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.560957 7577 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.563589 11086 tablet_replica.cc:330] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd: stopping tablet replica
I20240501 08:00:18.563742 11086 raft_consensus.cc:2229] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:18.563988 11086 raft_consensus.cc:2258] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:18.563920 7169 catalog_manager.cc:5484] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 6 to 7, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.565009 9187 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.566659 7447 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.567752 9318 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.568753 11094 tablet_replica.cc:330] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e: stopping tablet replica
I20240501 08:00:18.569108 11094 raft_consensus.cc:2229] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:18.569407 11094 pending_rounds.cc:62] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e: Trying to abort 1 pending ops.
I20240501 08:00:18.569485 11094 pending_rounds.cc:68] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e: Aborting op as it isn't in flight: id { term: 1 index: 11 } timestamp: 7022798514177613824 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "3abb5202d8d24d9b9a67f7a87b7300a6" old_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } } new_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } } }
I20240501 08:00:18.569646 11094 raft_consensus.cc:2894] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Aborting config change with OpId 1.11: Aborted: Op aborted
I20240501 08:00:18.569718 11094 raft_consensus.cc:2866] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 08:00:18.569762 11094 raft_consensus.cc:2258] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:18.570937 11094 ts_tablet_manager.cc:1875] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:18.572288 9055 raft_consensus.cc:1203] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:00:18.573450 10335 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.574527 10080 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:18.574824 11072 raft_consensus.cc:1050] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 to VOTER
I20240501 08:00:18.575053 9318 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:18.575285 11072 consensus_queue.cc:237] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.576027 9449 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.578042 7175 catalog_manager.cc:5484] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:00:18.579347 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81ccdd855b584d65860fd7a5777920a3. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.579651 9187 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.579866 7579 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.580004 8678 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.580713 7447 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.580731 7448 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:18.581799 7710 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:18.583138 11072 raft_consensus.cc:1050] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 to VOTER
I20240501 08:00:18.583621 11072 consensus_queue.cc:237] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.583809 7166 catalog_manager.cc:5484] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e reported cstate change: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.584538 7315 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.585253 9055 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.585878 8678 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.586112 7317 consensus_queue.cc:237] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:18.587172 9449 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.587630 8678 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.588872 9319 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.589388 7317 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.590157 10335 raft_consensus.cc:1013] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: attempt to promote peer 1807c0c2e7d04b20b0cda311c42eef8e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:00:18.590461 11073 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.590632 7448 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.591768 7579 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.592240 11073 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.592576 11040 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.592689 8678 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.592960 10974 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.594972 8678 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.595593 10974 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:18.597111 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5970772c02d04e0b95a2f2feb7dcd655 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.597470 7448 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:18.598045 9187 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.598737 7166 catalog_manager.cc:5484] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 5 to 6, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:18.600077 7577 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.600646 11088 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.600847 9318 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.601392 11088 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.602272 7688 tablet_service.cc:1514] Processing DeleteTablet for tablet 5970772c02d04e0b95a2f2feb7dcd655 with delete_type TABLET_DATA_TOMBSTONED (TS 09a7b6e2731a474b80411efb62323604 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49464
I20240501 08:00:18.602205 9318 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:18.602459 11058 tablet_replica.cc:330] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: stopping tablet replica
I20240501 08:00:18.602581 11058 raft_consensus.cc:2229] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:18.602733 11073 raft_consensus.cc:1050] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 to VOTER
I20240501 08:00:18.602733 11058 raft_consensus.cc:2258] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:18.603199 11073 consensus_queue.cc:237] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.603458 9057 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.604596 11058 ts_tablet_manager.cc:1875] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:18.604739 7448 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.604995 9449 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.605629 11088 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.606333 9057 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.606791 11088 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.606827 11058 ts_tablet_manager.cc:1888] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:18.607380 11058 log.cc:1196] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5970772c02d04e0b95a2f2feb7dcd655
I20240501 08:00:18.608402 7162 catalog_manager.cc:4830] TS 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789): tablet 5970772c02d04e0b95a2f2feb7dcd655 (table rebalance_test_table_2 [id=d578050a40e0474484896c98e4da9f5a]) successfully deleted
I20240501 08:00:18.608994 7576 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.609037 10335 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.609387 10870 raft_consensus.cc:1050] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e to VOTER
I20240501 08:00:18.609812 10870 consensus_queue.cc:237] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:18.610996 9450 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.611349 9057 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.611747 9055 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.612114 10907 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.614120 7315 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.614127 7175 catalog_manager.cc:5484] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7 reported cstate change: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.614602 10558 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.614672 7710 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.614706 9450 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.615432 10558 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.615948 7315 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.616125 11088 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.616571 7577 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.618223 10558 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.622615 11088 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.625137 7315 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.630667 7316 consensus_queue.cc:237] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.631501 7168 catalog_manager.cc:5484] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 reported cstate change: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.633070 9187 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.633790 10306 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:00:18.634531 7250 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02b42b5fd3d440ea950b35ed461b8e12. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:18.635979 7162 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 81ccdd855b584d65860fd7a5777920a3 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 7 but the committed config has opid_index of 8
I20240501 08:00:18.636775 9057 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.637292 10306 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.637852 7708 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.638403 10323 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.638602 10974 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.638994 10974 raft_consensus.cc:1013] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: attempt to promote peer d8aa5ac376a54e1f8417e5af11d28fa7: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:00:18.640197 7709 consensus_queue.cc:237] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.640304 11094 ts_tablet_manager.cc:1888] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240501 08:00:18.640421 11094 log.cc:1196] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 1807c0c2e7d04b20b0cda311c42eef8e: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/3abb5202d8d24d9b9a67f7a87b7300a6
I20240501 08:00:18.640679 9057 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.641674 7160 catalog_manager.cc:4830] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): tablet 3abb5202d8d24d9b9a67f7a87b7300a6 (table rebalance_test_table_0 [id=61deeb642b0b4f06a2ef4b1a391fdcb9]) successfully deleted
I20240501 08:00:18.641553 7577 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.642802 9449 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.643496 7315 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.644382 9447 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.646838 11088 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.647241 11088 raft_consensus.cc:1050] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e to VOTER
I20240501 08:00:18.647269 7174 catalog_manager.cc:5484] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.647585 11088 consensus_queue.cc:237] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.650694 9054 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.651142 9054 raft_consensus.cc:1203] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:00:18.651708 9057 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.652489 10335 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.652773 10335 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.653363 9449 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.653928 8678 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.655072 8678 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.655244 9528 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.656100 9318 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.657541 9054 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.658326 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1e44704a9b9c46cf954d8e6937983841 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.659422 9447 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.659608 7316 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.659896 7708 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.660205 7174 catalog_manager.cc:5484] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 5 to 6, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:18.662931 7172 catalog_manager.cc:5484] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 7 to 8, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:18.663960 7447 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.666632 7558 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e44704a9b9c46cf954d8e6937983841 with delete_type TABLET_DATA_TOMBSTONED (TS 31b02f06be8a4455b94b1254d77005bd not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:33196
I20240501 08:00:18.667552 7708 consensus_queue.cc:237] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.668292 10974 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.669974 9449 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.670432 10335 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:18.670302 11108 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.671471 7168 catalog_manager.cc:5484] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:18.672168 10335 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.673478 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1e44704a9b9c46cf954d8e6937983841 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.673656 9054 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.674158 9447 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.675495 11086 ts_tablet_manager.cc:1875] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:18.676157 9449 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.677078 7168 catalog_manager.cc:5484] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 reported cstate change: config changed from index 6 to 7, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.677706 9054 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.679807 9528 raft_consensus.cc:1050] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 to VOTER
I20240501 08:00:18.680862 7708 consensus_queue.cc:237] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.681295 9054 raft_consensus.cc:1261] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.681777 11088 consensus_queue.cc:1035] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:18.681955 10558 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:18.682824 9056 raft_consensus.cc:2932] T 1e44704a9b9c46cf954d8e6937983841 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.684137 9449 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.684150 7317 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:18.684556 11072 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.685341 9056 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.685876 10264 raft_consensus.cc:1050] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 to VOTER
I20240501 08:00:18.686136 11072 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.686316 10264 consensus_queue.cc:237] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.686573 9056 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.688941 7708 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.689323 10974 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.689582 7173 catalog_manager.cc:5484] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 8 to 9, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:18.689961 7317 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.690567 7317 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.690522 9449 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:18.691411 7447 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.691659 11072 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.692512 10974 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.692765 10912 tablet_bootstrap.cc:492] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.693032 10912 tablet_bootstrap.cc:492] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap complete.
I20240501 08:00:18.693259 7317 raft_consensus.cc:858] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
W20240501 08:00:18.693331 7317 raft_consensus.cc:1565] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 8' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.693830 7317 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.694397 7172 catalog_manager.cc:5484] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:18.695092 7297 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e44704a9b9c46cf954d8e6937983841 with delete_type TABLET_DATA_TOMBSTONED (TS b4f8020266b5455b872a2973fc396578 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35594
I20240501 08:00:18.695374 10923 tablet_replica.cc:330] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578: stopping tablet replica
I20240501 08:00:18.695511 10923 raft_consensus.cc:2229] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:18.695674 10923 raft_consensus.cc:2258] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:18.695410 10912 ts_tablet_manager.cc:1367] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent bootstrapping tablet: real 0.185s user 0.027s sys 0.001s
I20240501 08:00:18.695698 11072 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.695679 9528 consensus_queue.cc:237] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.696707 10912 raft_consensus.cc:348] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.697742 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4dde9710a5eb4baa881d9dd04affe184 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.697917 10923 ts_tablet_manager.cc:1875] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:18.698621 9054 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.698972 9449 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.700222 10923 ts_tablet_manager.cc:1888] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:00:18.700295 10923 log.cc:1196] T 1e44704a9b9c46cf954d8e6937983841 P b4f8020266b5455b872a2973fc396578: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/1e44704a9b9c46cf954d8e6937983841
I20240501 08:00:18.700889 7162 catalog_manager.cc:4830] TS b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): tablet 1e44704a9b9c46cf954d8e6937983841 (table rebalance_test_table_9 [id=15bea2064e1c414fb23182f6bc324bfd]) successfully deleted
I20240501 08:00:18.700911 9528 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.701167 10912 raft_consensus.cc:729] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1807c0c2e7d04b20b0cda311c42eef8e, State: Initialized, Role: LEARNER
I20240501 08:00:18.701296 10912 consensus_queue.cc:260] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.701496 7576 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.701545 7173 catalog_manager.cc:5484] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 7 to 8, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:18.701782 9056 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.702226 10912 ts_tablet_manager.cc:1398] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 08:00:18.702539 9054 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.702842 9528 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:00:18.705359 7250 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02b42b5fd3d440ea950b35ed461b8e12. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.706192 9319 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:18.707245 7317 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.708191 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-cd07a00b6817400f86bf8c32bd592d8b received from {username='slave'} at 127.0.99.136:42359
I20240501 08:00:18.708272 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-cd07a00b6817400f86bf8c32bd592d8b on tablet cd07a00b6817400f86bf8c32bd592d8b with peer 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:18.708680 7708 consensus_queue.cc:237] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.709738 7558 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dde9710a5eb4baa881d9dd04affe184 with delete_type TABLET_DATA_TOMBSTONED (TS 31b02f06be8a4455b94b1254d77005bd not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:33196
I20240501 08:00:18.710585 7576 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.710762 9449 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.711242 10323 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:18.711422 9528 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.712186 7576 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.713557 7447 raft_consensus.cc:858] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:00:18.713654 7447 raft_consensus.cc:1565] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:00:18.714205 7250 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e178f7c3c17c4e1ca3d16b6441f76e77. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.714102 7447 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.714558 9054 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.716223 10870 raft_consensus.cc:1050] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 to VOTER
I20240501 08:00:18.716620 10870 consensus_queue.cc:237] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:18.717005 9186 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.717589 11040 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.718305 9055 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.718704 10907 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.718863 10306 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.719658 9055 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:18.719676 9186 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.721130 9186 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.722748 7709 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.723062 11072 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
I20240501 08:00:18.724287 7446 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.724979 9449 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.725608 11116 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.725437 10323 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.726455 7174 catalog_manager.cc:5484] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.727562 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4dde9710a5eb4baa881d9dd04affe184 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.728106 9449 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.728384 9056 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.730671 7166 catalog_manager.cc:5484] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e reported cstate change: config changed from index 8 to 9, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.734140 7317 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.734938 10558 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.735024 10742 ts_tablet_manager.cc:902] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:18.735532 10742 tablet_copy_client.cc:308] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:18.735883 7297 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dde9710a5eb4baa881d9dd04affe184 with delete_type TABLET_DATA_TOMBSTONED (TS b4f8020266b5455b872a2973fc396578 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35594
I20240501 08:00:18.736049 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet 1088a116ccec4020aec562a69791a188 from peer 3b69c5d799e14e438df760453438aa75 ({username='slave'} at 127.0.99.134:33617)
I20240501 08:00:18.736137 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet 1088a116ccec4020aec562a69791a188 from peer 3b69c5d799e14e438df760453438aa75 at {username='slave'} at 127.0.99.134:33617: session id = 3b69c5d799e14e438df760453438aa75-1088a116ccec4020aec562a69791a188
I20240501 08:00:18.736203 10923 tablet_replica.cc:330] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: stopping tablet replica
I20240501 08:00:18.736054 9054 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.736332 10923 raft_consensus.cc:2229] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:18.736429 10558 raft_consensus.cc:1050] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e to VOTER
I20240501 08:00:18.736518 10923 raft_consensus.cc:2258] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:18.736881 10558 consensus_queue.cc:237] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.737140 7336 tablet_copy_source_session.cc:205] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:18.737259 10923 ts_tablet_manager.cc:1875] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:18.738582 10742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1088a116ccec4020aec562a69791a188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:18.738725 7709 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.739136 7446 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.739328 10559 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.739508 10923 ts_tablet_manager.cc:1888] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:00:18.739574 10923 log.cc:1196] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4dde9710a5eb4baa881d9dd04affe184
I20240501 08:00:18.739676 10559 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.740953 10742 tablet_copy_client.cc:783] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:18.741087 10742 tablet_copy_client.cc:647] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:18.741147 9318 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.741322 9186 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.741345 9449 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.741557 10264 raft_consensus.cc:1050] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER 3b69c5d799e14e438df760453438aa75 to VOTER
I20240501 08:00:18.741562 10870 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.742022 10264 consensus_queue.cc:237] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.743263 11086 ts_tablet_manager.cc:1888] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 08:00:18.743343 10870 raft_consensus.cc:1013] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: attempt to promote peer 1c8cb9b1059a47cc822ca13a2b931776: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:00:18.743466 11086 log.cc:1196] T 3abb5202d8d24d9b9a67f7a87b7300a6 P 31b02f06be8a4455b94b1254d77005bd: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3abb5202d8d24d9b9a67f7a87b7300a6
I20240501 08:00:18.743564 10870 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.743662 10559 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.744053 11086 tablet_replica.cc:330] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd: stopping tablet replica
I20240501 08:00:18.744310 7159 catalog_manager.cc:4830] TS 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861): tablet 3abb5202d8d24d9b9a67f7a87b7300a6 (table rebalance_test_table_0 [id=61deeb642b0b4f06a2ef4b1a391fdcb9]) successfully deleted
I20240501 08:00:18.745061 9187 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.745061 9447 raft_consensus.cc:1203] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 08:00:18.745085 11086 raft_consensus.cc:2229] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:18.745440 11086 raft_consensus.cc:2258] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:18.745802 11040 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.745867 7707 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.746562 11086 ts_tablet_manager.cc:1875] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:18.746742 11040 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.748191 10912 ts_tablet_manager.cc:902] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e: Initiating tablet copy from peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:18.748692 10912 tablet_copy_client.cc:308] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40707
I20240501 08:00:18.749434 11040 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.749540 10559 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.749758 7467 tablet_copy_service.cc:140] P d6b4be7bbd1649d0b7c5e8efe81f759f: Received BeginTabletCopySession request for tablet ffc23756b44d4d588e0a8175075d9bc6 from peer 1807c0c2e7d04b20b0cda311c42eef8e ({username='slave'} at 127.0.99.136:53469)
I20240501 08:00:18.749855 7467 tablet_copy_service.cc:161] P d6b4be7bbd1649d0b7c5e8efe81f759f: Beginning new tablet copy session on tablet ffc23756b44d4d588e0a8175075d9bc6 from peer 1807c0c2e7d04b20b0cda311c42eef8e at {username='slave'} at 127.0.99.136:53469: session id = 1807c0c2e7d04b20b0cda311c42eef8e-ffc23756b44d4d588e0a8175075d9bc6
I20240501 08:00:18.750347 7448 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.750455 11040 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.750833 7467 tablet_copy_source_session.cc:205] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:18.751684 10912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc23756b44d4d588e0a8175075d9bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:18.751789 7448 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.752455 9187 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.753582 7707 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.754828 9057 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.757061 7317 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.757275 7167 catalog_manager.cc:5484] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 reported cstate change: config changed from index 7 to 8, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.758003 9056 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.758527 9528 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.759668 9057 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.759853 9449 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.761010 10742 tablet_copy_client.cc:523] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:18.762156 10870 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.762534 10742 tablet_bootstrap.cc:492] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75: Bootstrap starting.
I20240501 08:00:18.762552 11075 raft_consensus.cc:1050] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f: attempting to promote NON_VOTER 3b69c5d799e14e438df760453438aa75 to VOTER
I20240501 08:00:18.762943 11075 consensus_queue.cc:237] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:18.763712 11142 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.764964 7707 raft_consensus.cc:1261] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.765015 9187 raft_consensus.cc:1261] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.765409 11074 consensus_queue.cc:1035] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.765311 9319 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.765578 10383 consensus_queue.cc:1035] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.766320 7162 catalog_manager.cc:4830] TS b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): tablet 4dde9710a5eb4baa881d9dd04affe184 (table rebalance_test_table_4 [id=ae84a8bf8ac745fb982412e9b194d7fc]) successfully deleted
I20240501 08:00:18.766981 7579 raft_consensus.cc:1261] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.767478 10907 raft_consensus.cc:1050] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 to VOTER
I20240501 08:00:18.767424 9449 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.767890 10907 consensus_queue.cc:237] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.753896 10912 tablet_copy_client.cc:783] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:18.769307 10912 tablet_copy_client.cc:647] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:18.770663 7173 catalog_manager.cc:5484] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.771255 9449 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.771457 7445 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.771567 7166 catalog_manager.cc:5484] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 reported cstate change: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.772101 9188 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.773550 10559 raft_consensus.cc:1050] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 to VOTER
I20240501 08:00:18.774049 10559 consensus_queue.cc:237] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.774873 11086 ts_tablet_manager.cc:1888] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:00:18.774957 11086 log.cc:1196] T 1e44704a9b9c46cf954d8e6937983841 P 31b02f06be8a4455b94b1254d77005bd: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1e44704a9b9c46cf954d8e6937983841
I20240501 08:00:18.775032 10559 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.775419 11086 tablet_replica.cc:330] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd: stopping tablet replica
I20240501 08:00:18.775550 11086 raft_consensus.cc:2229] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:18.776338 7317 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.776418 7159 catalog_manager.cc:4830] TS 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861): tablet 1e44704a9b9c46cf954d8e6937983841 (table rebalance_test_table_9 [id=15bea2064e1c414fb23182f6bc324bfd]) successfully deleted
I20240501 08:00:18.777137 10559 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.777626 9054 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.778132 7445 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.778843 7446 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.779767 11074 consensus_queue.cc:1035] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.780246 9318 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.780475 10558 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.780743 10558 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.780826 10912 tablet_copy_client.cc:523] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:18.781013 10558 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.781667 11086 pending_rounds.cc:62] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd: Trying to abort 1 pending ops.
I20240501 08:00:18.782145 11086 pending_rounds.cc:68] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7022798514517078016 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "4dde9710a5eb4baa881d9dd04affe184" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } } }
I20240501 08:00:18.782298 11086 raft_consensus.cc:2894] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240501 08:00:18.782366 11086 raft_consensus.cc:2866] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 08:00:18.782423 11086 raft_consensus.cc:2258] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:18.782649 10912 tablet_bootstrap.cc:492] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap starting.
I20240501 08:00:18.782799 7708 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.782761 7446 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.783483 10558 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.784348 7708 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:18.785692 9319 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.785877 7708 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.786063 11086 ts_tablet_manager.cc:1875] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:18.786926 10907 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.788427 11086 ts_tablet_manager.cc:1888] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:00:18.788486 11086 log.cc:1196] T 4dde9710a5eb4baa881d9dd04affe184 P 31b02f06be8a4455b94b1254d77005bd: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4dde9710a5eb4baa881d9dd04affe184
I20240501 08:00:18.788982 7159 catalog_manager.cc:4830] TS 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861): tablet 4dde9710a5eb4baa881d9dd04affe184 (table rebalance_test_table_4 [id=ae84a8bf8ac745fb982412e9b194d7fc]) successfully deleted
I20240501 08:00:18.789714 9188 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.789680 7316 consensus_queue.cc:237] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.789777 7173 catalog_manager.cc:5484] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 6 to 7, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER, now has a pending config: VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131), VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132), VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130), NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134), VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135), VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136). New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } } pending_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.790755 9447 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.791556 9054 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.791909 10559 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.791991 10264 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.793335 9188 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.793689 11040 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.795506 9057 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.796479 10870 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.797416 10870 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.797587 10559 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.798307 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e178f7c3c17c4e1ca3d16b6441f76e77 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.800614 10907 raft_consensus.cc:1013] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: attempt to promote peer 3b69c5d799e14e438df760453438aa75: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:00:18.801019 9319 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.803014 7448 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.804193 7166 catalog_manager.cc:5484] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 reported cstate change: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131), VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132), VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130), NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134), VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135), VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136). New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.808051 10559 raft_consensus.cc:1050] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER 3b69c5d799e14e438df760453438aa75 to VOTER
I20240501 08:00:18.808559 10559 consensus_queue.cc:237] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.810900 9187 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.811463 7579 consensus_queue.cc:237] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.811558 7166 catalog_manager.cc:5484] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 reported cstate change: config changed from index 6 to 7, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.814049 7316 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.814433 7708 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.814460 9187 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.814806 9054 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.815101 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet e178f7c3c17c4e1ca3d16b6441f76e77 with delete_type TABLET_DATA_TOMBSTONED (TS d6b4be7bbd1649d0b7c5e8efe81f759f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45222
I20240501 08:00:18.815680 11154 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:18.816067 11154 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.816514 11074 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:18.817337 11155 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.818552 11155 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.819093 7708 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:18.820711 11199 tablet_replica.cc:330] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f: stopping tablet replica
I20240501 08:00:18.821671 9187 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
W20240501 08:00:18.822279 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8311d0921e9e417090e441884615525a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.822695 7576 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:18.823271 9319 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.823971 7448 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.824263 11199 raft_consensus.cc:2229] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:18.824548 11199 raft_consensus.cc:2258] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:18.824522 10870 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:18.824844 10870 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:18.825651 7172 catalog_manager.cc:5484] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:18.828842 10558 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.829803 9188 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.830044 7448 consensus_queue.cc:237] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:18.831105 9187 raft_consensus.cc:1261] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.831558 11075 consensus_queue.cc:1035] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.831771 7576 raft_consensus.cc:1261] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.832298 9319 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.832866 7576 consensus_queue.cc:237] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:18.832986 11075 consensus_queue.cc:1035] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.834385 9057 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.834589 7315 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.835006 10907 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.835919 10558 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.835882 7173 catalog_manager.cc:5484] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 reported cstate change: config changed from index 8 to 9, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.837076 10907 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.838174 10870 raft_consensus.cc:1013] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: attempt to promote peer d8aa5ac376a54e1f8417e5af11d28fa7: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:00:18.839073 9447 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.839507 7705 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.839536 10907 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.839936 7707 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.840720 9188 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.841459 7316 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.841456 9057 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.841888 10558 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.842835 7315 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.844556 7173 catalog_manager.cc:5484] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 7 to 8, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.845036 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cc6f7dcc930e4937865d37cb3c3bb3d2 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.845495 9318 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.845532 7447 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.845717 10558 raft_consensus.cc:1050] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e to VOTER
I20240501 08:00:18.846091 10558 consensus_queue.cc:237] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.842351 11204 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.844552 10870 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.849134 9318 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.849161 7447 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.849648 9447 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.849767 7172 catalog_manager.cc:5484] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 reported cstate change: config changed from index 6 to 7, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:18.849978 9448 raft_consensus.cc:1203] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:00:18.850450 10907 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.850937 7705 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.851433 7707 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.852490 7705 raft_consensus.cc:858] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
W20240501 08:00:18.852548 7705 raft_consensus.cc:1565] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.852788 7705 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.854194 9057 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.855443 9447 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:18.856763 7688 tablet_service.cc:1514] Processing DeleteTablet for tablet cc6f7dcc930e4937865d37cb3c3bb3d2 with delete_type TABLET_DATA_TOMBSTONED (TS 09a7b6e2731a474b80411efb62323604 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:49464
I20240501 08:00:18.856963 11058 tablet_replica.cc:330] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: stopping tablet replica
I20240501 08:00:18.857088 11058 raft_consensus.cc:2229] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:18.857256 11058 raft_consensus.cc:2258] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:18.858002 11058 ts_tablet_manager.cc:1875] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:18.858326 7166 catalog_manager.cc:5484] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 reported cstate change: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.858583 10907 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.858940 10907 raft_consensus.cc:1050] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 to VOTER
I20240501 08:00:18.859417 10907 consensus_queue.cc:237] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.860062 11058 ts_tablet_manager.cc:1888] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:00:18.860126 11058 log.cc:1196] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/cc6f7dcc930e4937865d37cb3c3bb3d2
I20240501 08:00:18.860908 7162 catalog_manager.cc:4830] TS 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789): tablet cc6f7dcc930e4937865d37cb3c3bb3d2 (table rebalance_test_table_2 [id=d578050a40e0474484896c98e4da9f5a]) successfully deleted
I20240501 08:00:18.861218 9057 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.861569 7316 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.861636 10870 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:00:18.861748 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8311d0921e9e417090e441884615525a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.862375 7707 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.863134 11208 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.863440 11208 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.863895 11208 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.863834 7448 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.866292 7447 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.866855 9447 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.867568 9188 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:18.868229 10870 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.868577 10870 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.869482 9447 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.869509 9188 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.870826 9318 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.871830 7167 catalog_manager.cc:5484] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 7 to 8, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.873060 7705 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.874388 9057 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.876036 10907 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.878360 7316 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.878927 10147 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:18.881008 7172 catalog_manager.cc:5484] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.881060 7707 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.882011 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f647cfc562794e65a05918b241014d72 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.883246 9188 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:18.883845 7576 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:18.885603 10907 raft_consensus.cc:1050] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER 3b69c5d799e14e438df760453438aa75 to VOTER
I20240501 08:00:18.886057 10907 consensus_queue.cc:237] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.886173 7166 catalog_manager.cc:5484] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 5 to 6, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:18.885427 10742 tablet_bootstrap.cc:492] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.886714 10742 tablet_bootstrap.cc:492] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75: Bootstrap complete.
I20240501 08:00:18.886869 10742 ts_tablet_manager.cc:1367] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75: Time spent bootstrapping tablet: real 0.124s user 0.026s sys 0.003s
I20240501 08:00:18.887864 9447 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.887881 9318 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.888247 11205 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:18.888485 7447 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.888851 11205 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:18.892644 10742 raft_consensus.cc:348] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:18.892976 11209 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:18.893738 7688 tablet_service.cc:1514] Processing DeleteTablet for tablet f647cfc562794e65a05918b241014d72 with delete_type TABLET_DATA_TOMBSTONED (TS 09a7b6e2731a474b80411efb62323604 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49464
I20240501 08:00:18.893959 11058 tablet_replica.cc:330] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604: stopping tablet replica
I20240501 08:00:18.893393 10742 raft_consensus.cc:729] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b69c5d799e14e438df760453438aa75, State: Initialized, Role: LEARNER
I20240501 08:00:18.894163 11058 raft_consensus.cc:2229] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:18.894205 10742 consensus_queue.cc:260] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:18.894315 11058 raft_consensus.cc:2258] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:18.894408 9188 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.894610 10742 ts_tablet_manager.cc:1398] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 08:00:18.895155 11155 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:18.895876 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session 3b69c5d799e14e438df760453438aa75-1088a116ccec4020aec562a69791a188 received from {username='slave'} at 127.0.99.134:33617
I20240501 08:00:18.895947 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session 3b69c5d799e14e438df760453438aa75-1088a116ccec4020aec562a69791a188 on tablet 1088a116ccec4020aec562a69791a188 with peer 3b69c5d799e14e438df760453438aa75
I20240501 08:00:18.896775 11142 raft_consensus.cc:1050] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER 3b69c5d799e14e438df760453438aa75 to VOTER
I20240501 08:00:18.897179 11142 consensus_queue.cc:237] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:18.897500 7579 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.898732 7707 raft_consensus.cc:1261] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.899039 9187 raft_consensus.cc:1261] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.899145 10306 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.900292 10306 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.901926 11204 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.902367 11199 ts_tablet_manager.cc:1875] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:18.904166 7576 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.904281 9318 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.904834 11199 ts_tablet_manager.cc:1888] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:00:18.904915 11199 log.cc:1196] T e178f7c3c17c4e1ca3d16b6441f76e77 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e178f7c3c17c4e1ca3d16b6441f76e77
I20240501 08:00:18.906345 7159 catalog_manager.cc:4830] TS d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707): tablet e178f7c3c17c4e1ca3d16b6441f76e77 (table rebalance_test_table_5 [id=2427aea1b9a6436ca7bd1796a85634ff]) successfully deleted
I20240501 08:00:18.907742 9447 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.908955 7448 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.910483 7171 catalog_manager.cc:5484] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 reported cstate change: config changed from index 8 to 9, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.910640 9187 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
W20240501 08:00:18.911468 7250 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02b42b5fd3d440ea950b35ed461b8e12. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:18.911578 7250 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e178f7c3c17c4e1ca3d16b6441f76e77. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.912525 11143 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.913463 11143 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.913939 10323 raft_consensus.cc:1050] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 to VOTER
I20240501 08:00:18.914394 10323 consensus_queue.cc:237] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:18.916622 9318 raft_consensus.cc:1261] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.917577 11109 consensus_queue.cc:1035] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.918908 7447 raft_consensus.cc:1261] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.919631 7447 raft_consensus.cc:858] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:00:18.919705 7447 raft_consensus.cc:1565] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 8' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.920342 7447 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.921249 10499 ts_tablet_manager.cc:902] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776: Initiating tablet copy from peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:18.921738 10499 tablet_copy_client.cc:308] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40707
I20240501 08:00:18.922537 7467 tablet_copy_service.cc:140] P d6b4be7bbd1649d0b7c5e8efe81f759f: Received BeginTabletCopySession request for tablet 5d18a2918c75467a9fe59979ce8936cd from peer 1c8cb9b1059a47cc822ca13a2b931776 ({username='slave'} at 127.0.99.135:41949)
I20240501 08:00:18.922618 7467 tablet_copy_service.cc:161] P d6b4be7bbd1649d0b7c5e8efe81f759f: Beginning new tablet copy session on tablet 5d18a2918c75467a9fe59979ce8936cd from peer 1c8cb9b1059a47cc822ca13a2b931776 at {username='slave'} at 127.0.99.135:41949: session id = 1c8cb9b1059a47cc822ca13a2b931776-5d18a2918c75467a9fe59979ce8936cd
I20240501 08:00:18.923485 7467 tablet_copy_source_session.cc:205] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:18.924499 10499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d18a2918c75467a9fe59979ce8936cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:18.929404 10499 tablet_copy_client.cc:783] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:18.929576 10499 tablet_copy_client.cc:647] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:18.929836 10912 tablet_bootstrap.cc:492] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.930702 10912 tablet_bootstrap.cc:492] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap complete.
I20240501 08:00:18.931022 10912 ts_tablet_manager.cc:1367] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent bootstrapping tablet: real 0.148s user 0.023s sys 0.004s
I20240501 08:00:18.932896 9450 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.933885 7579 raft_consensus.cc:1261] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.934572 10912 raft_consensus.cc:348] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.934777 10912 raft_consensus.cc:729] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1807c0c2e7d04b20b0cda311c42eef8e, State: Initialized, Role: LEARNER
I20240501 08:00:18.935324 10907 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:18.935717 7579 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.935843 10912 consensus_queue.cc:260] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.936187 10912 ts_tablet_manager.cc:1398] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:18.936841 7467 tablet_copy_service.cc:342] P d6b4be7bbd1649d0b7c5e8efe81f759f: Request end of tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-ffc23756b44d4d588e0a8175075d9bc6 received from {username='slave'} at 127.0.99.136:53469
I20240501 08:00:18.936923 7467 tablet_copy_service.cc:433] P d6b4be7bbd1649d0b7c5e8efe81f759f: ending tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-ffc23756b44d4d588e0a8175075d9bc6 on tablet ffc23756b44d4d588e0a8175075d9bc6 with peer 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:18.938541 11058 ts_tablet_manager.cc:1875] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:18.940102 7314 consensus_queue.cc:237] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:18.940789 11058 ts_tablet_manager.cc:1888] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:18.940877 11058 log.cc:1196] T f647cfc562794e65a05918b241014d72 P 09a7b6e2731a474b80411efb62323604: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f647cfc562794e65a05918b241014d72
I20240501 08:00:18.941494 7162 catalog_manager.cc:4830] TS 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789): tablet f647cfc562794e65a05918b241014d72 (table rebalance_test_table_7 [id=c7ddae9ad2184cb9bd097d68171c3fc4]) successfully deleted
I20240501 08:00:18.941687 10323 consensus_queue.cc:1035] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.941808 9450 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.943367 9450 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.944191 9188 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.944813 10264 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.946318 9318 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.946792 10264 raft_consensus.cc:1013] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: attempt to promote peer 3b69c5d799e14e438df760453438aa75: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:00:18.948848 10383 raft_consensus.cc:1050] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f: attempting to promote NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e to VOTER
I20240501 08:00:18.949071 10742 ts_tablet_manager.cc:902] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:18.949270 10383 consensus_queue.cc:237] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.949573 10742 tablet_copy_client.cc:308] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:00:18.950019 7728 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 07967f68a412411494c6dd1f47209009 from peer 3b69c5d799e14e438df760453438aa75 ({username='slave'} at 127.0.99.134:52489)
I20240501 08:00:18.950120 7728 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 07967f68a412411494c6dd1f47209009 from peer 3b69c5d799e14e438df760453438aa75 at {username='slave'} at 127.0.99.134:52489: session id = 3b69c5d799e14e438df760453438aa75-07967f68a412411494c6dd1f47209009
I20240501 08:00:18.950793 7707 raft_consensus.cc:1261] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.951123 7728 tablet_copy_source_session.cc:205] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:18.951190 10147 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.952335 10742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07967f68a412411494c6dd1f47209009. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:00:18.952428 7251 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1088a116ccec4020aec562a69791a188. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.952509 9450 raft_consensus.cc:1261] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.952862 10147 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.954264 7448 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.954574 10742 tablet_copy_client.cc:783] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:18.954733 10742 tablet_copy_client.cc:647] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:18.955045 7314 raft_consensus.cc:1261] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.955415 10499 tablet_copy_client.cc:523] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:18.955507 10323 raft_consensus.cc:1050] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER 3b69c5d799e14e438df760453438aa75 to VOTER
I20240501 08:00:18.955894 10323 consensus_queue.cc:237] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:18.955966 11143 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.957026 10499 tablet_bootstrap.cc:492] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap starting.
I20240501 08:00:18.957628 7448 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.957705 9188 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.958024 11072 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.958164 10335 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.959228 7314 consensus_queue.cc:237] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:18.959292 10264 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:18.959877 11072 consensus_queue.cc:1035] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.960690 9187 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.960822 7707 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.960968 9057 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:18.961369 10264 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:18.961650 10264 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:18.962972 10974 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:18.963758 10974 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:18.965173 9054 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:18.966511 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 02b42b5fd3d440ea950b35ed461b8e12 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.966804 7707 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:18.969722 9187 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:18.971060 7579 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.971375 7577 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:18.971818 11072 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.971953 7575 raft_consensus.cc:1261] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:18.973258 10147 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:18.968595 7173 catalog_manager.cc:5484] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 reported cstate change: config changed from index 8 to 9, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:18.974341 10742 tablet_copy_client.cc:523] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:18.975864 10742 tablet_bootstrap.cc:492] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75: Bootstrap starting.
I20240501 08:00:18.977627 9298 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b42b5fd3d440ea950b35ed461b8e12 with delete_type TABLET_DATA_TOMBSTONED (TS 1c8cb9b1059a47cc822ca13a2b931776 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40266
I20240501 08:00:18.978385 7579 consensus_queue.cc:237] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:18.978757 7578 consensus_queue.cc:237] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.981866 7446 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.982534 7577 consensus_queue.cc:237] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:18.982954 7313 consensus_queue.cc:237] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:18.984649 7705 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:18.985069 11116 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:00:18.987159 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8311d0921e9e417090e441884615525a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:18.987725 9319 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:18.987999 7707 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.988162 9318 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:18.988621 11209 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:18.988785 10907 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.988910 11209 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:18.990445 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:18.990818 9057 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:18.990818 11116 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.991200 10559 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:18.992190 9056 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.993588 9054 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:18.994071 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8311d0921e9e417090e441884615525a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.994580 9187 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:18.994887 7705 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.995895 11155 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:18.997212 10306 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:18.997572 10306 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:18.998257 7170 catalog_manager.cc:5484] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 6 to 7, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:18.998275 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21d76484f5dd499fb6fc3d534bffb779 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.998783 9318 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.000358 7705 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.001443 7297 tablet_service.cc:1514] Processing DeleteTablet for tablet 8311d0921e9e417090e441884615525a with delete_type TABLET_DATA_TOMBSTONED (TS b4f8020266b5455b872a2973fc396578 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35594
I20240501 08:00:19.001664 10923 tablet_replica.cc:330] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578: stopping tablet replica
I20240501 08:00:19.001667 7170 catalog_manager.cc:5484] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 reported cstate change: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.001793 10923 raft_consensus.cc:2229] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.002043 10923 raft_consensus.cc:2258] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.002975 10923 ts_tablet_manager.cc:1875] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.004050 11266 ts_tablet_manager.cc:902] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:19.004614 11266 tablet_copy_client.cc:308] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:00:19.005146 10923 ts_tablet_manager.cc:1888] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:00:19.005198 10923 log.cc:1196] T 8311d0921e9e417090e441884615525a P b4f8020266b5455b872a2973fc396578: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/8311d0921e9e417090e441884615525a
I20240501 08:00:19.005704 7162 catalog_manager.cc:4830] TS b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): tablet 8311d0921e9e417090e441884615525a (table rebalance_test_table_6 [id=22d40ca8e4714c2495974e28872c5687]) successfully deleted
W20240501 08:00:19.006173 7159 catalog_manager.cc:4797] TS 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): delete failed for tablet 02b42b5fd3d440ea950b35ed461b8e12 because the tablet was not found. No further retry: Not found: Tablet not found: 02b42b5fd3d440ea950b35ed461b8e12
I20240501 08:00:19.006578 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d76484f5dd499fb6fc3d534bffb779 with delete_type TABLET_DATA_TOMBSTONED (TS d6b4be7bbd1649d0b7c5e8efe81f759f not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45222
I20240501 08:00:19.006810 11199 tablet_replica.cc:330] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: stopping tablet replica
I20240501 08:00:19.006934 11199 raft_consensus.cc:2229] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.007122 11199 raft_consensus.cc:2258] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.007803 11199 ts_tablet_manager.cc:1875] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.008880 7728 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 4ea0db42a75a4879b3d3d608666dc2bf from peer 3b69c5d799e14e438df760453438aa75 ({username='slave'} at 127.0.99.134:52489)
I20240501 08:00:19.008968 7728 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 4ea0db42a75a4879b3d3d608666dc2bf from peer 3b69c5d799e14e438df760453438aa75 at {username='slave'} at 127.0.99.134:52489: session id = 3b69c5d799e14e438df760453438aa75-4ea0db42a75a4879b3d3d608666dc2bf
I20240501 08:00:19.009833 7728 tablet_copy_source_session.cc:205] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:19.009876 11199 ts_tablet_manager.cc:1888] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 08:00:19.009938 11199 log.cc:1196] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/21d76484f5dd499fb6fc3d534bffb779
I20240501 08:00:19.011202 7159 catalog_manager.cc:4830] TS d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707): tablet 21d76484f5dd499fb6fc3d534bffb779 (table rebalance_test_table_1 [id=ba8a985906c04117a440386d3011b936]) successfully deleted
I20240501 08:00:19.014205 11266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea0db42a75a4879b3d3d608666dc2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:19.014072 11143 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.014427 9187 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.015763 10912 ts_tablet_manager.cc:902] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:19.016263 10912 tablet_copy_client.cc:308] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:00:19.016673 11266 tablet_copy_client.cc:783] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:19.016733 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:19.016821 11266 tablet_copy_client.cc:647] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:19.017217 9056 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.018296 7170 catalog_manager.cc:5484] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 reported cstate change: config changed from index 9 to 10, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.021006 11225 ts_tablet_manager.cc:902] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:19.021199 7728 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 4ea0db42a75a4879b3d3d608666dc2bf from peer 1807c0c2e7d04b20b0cda311c42eef8e ({username='slave'} at 127.0.99.136:56099)
I20240501 08:00:19.021297 7728 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 4ea0db42a75a4879b3d3d608666dc2bf from peer 1807c0c2e7d04b20b0cda311c42eef8e at {username='slave'} at 127.0.99.136:56099: session id = 1807c0c2e7d04b20b0cda311c42eef8e-4ea0db42a75a4879b3d3d608666dc2bf
I20240501 08:00:19.021723 11225 tablet_copy_client.cc:308] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:19.022156 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7 from peer d8aa5ac376a54e1f8417e5af11d28fa7 ({username='slave'} at 127.0.99.133:50977)
I20240501 08:00:19.022255 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7 from peer d8aa5ac376a54e1f8417e5af11d28fa7 at {username='slave'} at 127.0.99.133:50977: session id = d8aa5ac376a54e1f8417e5af11d28fa7-3d8ae3c82c9d4c5c858fa3f9d5959eb7
I20240501 08:00:19.022358 7728 tablet_copy_source_session.cc:205] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:19.023155 7336 tablet_copy_source_session.cc:205] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:19.023211 10912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea0db42a75a4879b3d3d608666dc2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:19.024194 11225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:19.025636 10912 tablet_copy_client.cc:783] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:19.025784 10912 tablet_copy_client.cc:647] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:19.027027 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 02b42b5fd3d440ea950b35ed461b8e12 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.027977 7315 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:19.028205 7313 consensus_queue.cc:237] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:19.028381 7316 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:19.028674 7314 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:19.029711 9056 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:19.030339 11040 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:19.030403 9187 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:19.030989 11040 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:19.032625 7688 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b42b5fd3d440ea950b35ed461b8e12 with delete_type TABLET_DATA_TOMBSTONED (TS 09a7b6e2731a474b80411efb62323604 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:49464
I20240501 08:00:19.033134 11058 tablet_replica.cc:330] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604: stopping tablet replica
I20240501 08:00:19.033113 11040 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.033305 11058 raft_consensus.cc:2229] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.033551 11058 raft_consensus.cc:2258] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.034413 11058 ts_tablet_manager.cc:1875] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.036690 11058 ts_tablet_manager.cc:1888] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 08:00:19.036756 11058 log.cc:1196] T 02b42b5fd3d440ea950b35ed461b8e12 P 09a7b6e2731a474b80411efb62323604: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/02b42b5fd3d440ea950b35ed461b8e12
I20240501 08:00:19.037919 7162 catalog_manager.cc:4830] TS 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789): tablet 02b42b5fd3d440ea950b35ed461b8e12 (table rebalance_test_table_6 [id=22d40ca8e4714c2495974e28872c5687]) successfully deleted
I20240501 08:00:19.038162 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:19.038422 7446 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:19.038728 11040 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.039412 11207 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:19.039734 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:19.040601 7314 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.041975 7314 raft_consensus.cc:858] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
W20240501 08:00:19.042079 7314 raft_consensus.cc:1565] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 9' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:19.042053 9056 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.042294 7314 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.043221 9450 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:19.043634 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 02b42b5fd3d440ea950b35ed461b8e12 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.045033 7166 catalog_manager.cc:5484] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 reported cstate change: config changed from index 10 to 11, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.045260 10264 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.047116 7707 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.050148 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b42b5fd3d440ea950b35ed461b8e12 with delete_type TABLET_DATA_TOMBSTONED (TS d6b4be7bbd1649d0b7c5e8efe81f759f not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:45222
I20240501 08:00:19.050439 11199 tablet_replica.cc:330] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f: stopping tablet replica
I20240501 08:00:19.050319 11143 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.050617 11199 raft_consensus.cc:2229] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.050607 7173 catalog_manager.cc:5484] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:19.050804 11199 raft_consensus.cc:2258] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.051663 11199 ts_tablet_manager.cc:1875] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.053195 7313 consensus_queue.cc:237] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:19.053880 11199 ts_tablet_manager.cc:1888] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:00:19.053947 11199 log.cc:1196] T 02b42b5fd3d440ea950b35ed461b8e12 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/02b42b5fd3d440ea950b35ed461b8e12
I20240501 08:00:19.054500 7707 raft_consensus.cc:1261] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.054924 10974 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.055459 9184 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.057512 9186 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.058548 7173 catalog_manager.cc:5484] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 reported cstate change: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.059108 9184 raft_consensus.cc:1261] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.059993 10306 raft_consensus.cc:1050] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER 3b69c5d799e14e438df760453438aa75 to VOTER
I20240501 08:00:19.061354 9184 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.062619 9186 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:19.063243 9450 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:19.063620 11143 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.063797 10335 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.064215 9184 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:19.066210 9318 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.067198 7576 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.068926 11203 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:19.069093 7173 catalog_manager.cc:5484] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776 reported cstate change: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.069226 11203 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:19.069485 11203 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:19.071592 7314 raft_consensus.cc:858] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
W20240501 08:00:19.071667 7314 raft_consensus.cc:1565] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 10' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:19.071758 10080 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.071892 7314 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.073233 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.073601 11266 tablet_copy_client.cc:523] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:19.074016 7707 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.075037 11266 tablet_bootstrap.cc:492] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75: Bootstrap starting.
I20240501 08:00:19.075953 9184 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.078773 7159 catalog_manager.cc:4830] TS d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707): tablet 02b42b5fd3d440ea950b35ed461b8e12 (table rebalance_test_table_6 [id=22d40ca8e4714c2495974e28872c5687]) successfully deleted
I20240501 08:00:19.079599 7173 catalog_manager.cc:5484] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 3b69c5d799e14e438df760453438aa75 reported cstate change: config changed from index 5 to 6, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.082257 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626 with delete_type TABLET_DATA_TOMBSTONED (TS d6b4be7bbd1649d0b7c5e8efe81f759f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45222
I20240501 08:00:19.082541 11199 tablet_replica.cc:330] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f: stopping tablet replica
I20240501 08:00:19.082726 11199 raft_consensus.cc:2229] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.082922 11199 pending_rounds.cc:62] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f: Trying to abort 1 pending ops.
I20240501 08:00:19.083067 11199 pending_rounds.cc:68] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022798515801767936 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "b8fdc2b9c2ef44b0bfec59a1a2f6b626" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } } }
I20240501 08:00:19.083312 11199 raft_consensus.cc:2894] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 08:00:19.083470 11199 raft_consensus.cc:2866] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 08:00:19.083566 11199 raft_consensus.cc:2258] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.086069 10742 tablet_bootstrap.cc:492] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:19.086366 10742 tablet_bootstrap.cc:492] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75: Bootstrap complete.
I20240501 08:00:19.086544 10742 ts_tablet_manager.cc:1367] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75: Time spent bootstrapping tablet: real 0.111s user 0.027s sys 0.000s
I20240501 08:00:19.086618 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:19.087720 10306 consensus_queue.cc:237] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:19.087898 10499 tablet_bootstrap.cc:492] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:19.088146 10499 tablet_bootstrap.cc:492] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap complete.
I20240501 08:00:19.088299 10499 ts_tablet_manager.cc:1367] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776: Time spent bootstrapping tablet: real 0.131s user 0.023s sys 0.004s
I20240501 08:00:19.088686 10499 raft_consensus.cc:348] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:19.088791 10499 raft_consensus.cc:729] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c8cb9b1059a47cc822ca13a2b931776, State: Initialized, Role: LEARNER
I20240501 08:00:19.088876 10499 consensus_queue.cc:260] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:19.089679 10499 ts_tablet_manager.cc:1398] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:19.090202 7467 tablet_copy_service.cc:342] P d6b4be7bbd1649d0b7c5e8efe81f759f: Request end of tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-5d18a2918c75467a9fe59979ce8936cd received from {username='slave'} at 127.0.99.135:41949
I20240501 08:00:19.090296 7467 tablet_copy_service.cc:433] P d6b4be7bbd1649d0b7c5e8efe81f759f: ending tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-5d18a2918c75467a9fe59979ce8936cd on tablet 5d18a2918c75467a9fe59979ce8936cd with peer 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:19.091223 10742 raft_consensus.cc:348] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:19.091416 10742 raft_consensus.cc:729] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b69c5d799e14e438df760453438aa75, State: Initialized, Role: LEARNER
I20240501 08:00:19.091636 10742 consensus_queue.cc:260] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:19.091957 10742 ts_tablet_manager.cc:1398] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20240501 08:00:19.093370 7251 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1088a116ccec4020aec562a69791a188. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.094604 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session 3b69c5d799e14e438df760453438aa75-07967f68a412411494c6dd1f47209009 received from {username='slave'} at 127.0.99.134:52489
I20240501 08:00:19.094753 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session 3b69c5d799e14e438df760453438aa75-07967f68a412411494c6dd1f47209009 on tablet 07967f68a412411494c6dd1f47209009 with peer 3b69c5d799e14e438df760453438aa75
W20240501 08:00:19.098506 7250 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.130:40707, user_credentials={real_user=slave}} blocked reactor thread for 48385.3us
I20240501 08:00:19.101009 10147 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.101223 11072 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.102561 7448 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.104298 9450 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.107864 9319 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:19.107839 7448 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.109429 7447 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:19.109582 11296 ts_tablet_manager.cc:902] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:19.109660 7576 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:19.110215 11296 tablet_copy_client.cc:308] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:00:19.110651 10912 tablet_copy_client.cc:523] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:19.110724 7728 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 81ccdd855b584d65860fd7a5777920a3 from peer 1807c0c2e7d04b20b0cda311c42eef8e ({username='slave'} at 127.0.99.136:56099)
I20240501 08:00:19.110821 7728 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 81ccdd855b584d65860fd7a5777920a3 from peer 1807c0c2e7d04b20b0cda311c42eef8e at {username='slave'} at 127.0.99.136:56099: session id = 1807c0c2e7d04b20b0cda311c42eef8e-81ccdd855b584d65860fd7a5777920a3
I20240501 08:00:19.110831 9186 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.111712 7728 tablet_copy_source_session.cc:205] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:19.112092 10912 tablet_bootstrap.cc:492] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap starting.
I20240501 08:00:19.112290 7576 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.113843 11206 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:19.115032 11206 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.116413 11040 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.116842 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bde42e485c4b49ce8acd694a25ba683a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.117271 11040 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:19.117257 9319 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.117427 7313 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.117758 7707 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.118295 11143 raft_consensus.cc:1050] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 to VOTER
I20240501 08:00:19.119031 11296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ccdd855b584d65860fd7a5777920a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:19.119758 11143 consensus_queue.cc:237] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:19.119989 7173 catalog_manager.cc:5484] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 reported cstate change: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.121001 11040 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.122661 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet bde42e485c4b49ce8acd694a25ba683a with delete_type TABLET_DATA_TOMBSTONED (TS d6b4be7bbd1649d0b7c5e8efe81f759f not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45222
I20240501 08:00:19.123039 7166 catalog_manager.cc:5484] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 4 to 5, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:19.123411 7166 catalog_manager.cc:5484] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.123654 9319 raft_consensus.cc:1261] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.124003 9318 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.126211 9187 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:19.127449 7447 consensus_queue.cc:237] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:19.128868 11199 ts_tablet_manager.cc:1875] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.129415 11040 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:19.130133 7577 raft_consensus.cc:1261] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.130174 7705 raft_consensus.cc:1261] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.130436 7576 raft_consensus.cc:858] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
W20240501 08:00:19.130513 7576 raft_consensus.cc:1565] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:19.130805 7576 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.131050 9449 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.131933 11040 consensus_queue.cc:1035] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.132201 11074 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.132567 11040 consensus_queue.cc:1035] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.133291 11040 raft_consensus.cc:1013] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: attempt to promote peer 1c8cb9b1059a47cc822ca13a2b931776: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:00:19.133751 10080 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.135550 9450 raft_consensus.cc:1261] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.135957 7447 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.136054 9319 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.137089 10383 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.138779 7166 catalog_manager.cc:5484] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 6 to 7, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.139277 10383 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.139541 11225 tablet_copy_client.cc:783] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:19.139721 11225 tablet_copy_client.cc:647] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:19.139643 7314 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.140918 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dd17760cd5c2454db97610b2f71c0ee2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.141599 7705 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.143432 9447 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.144388 11296 tablet_copy_client.cc:783] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:19.144547 11296 tablet_copy_client.cc:647] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:19.141281 9187 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.145290 11143 raft_consensus.cc:1050] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 to VOTER
I20240501 08:00:19.146703 11143 consensus_queue.cc:237] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:19.147372 7575 consensus_queue.cc:237] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:19.147572 9188 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
W20240501 08:00:19.148855 7251 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1088a116ccec4020aec562a69791a188. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.148963 7707 raft_consensus.cc:1261] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.149546 9450 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.149788 10264 consensus_queue.cc:1035] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.150925 9317 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:19.150969 9319 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:19.151463 9187 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:19.151453 10558 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:19.151844 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:19.151829 10264 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:19.152273 7173 catalog_manager.cc:5484] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e reported cstate change: config changed from index 5 to 6, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.152951 7447 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:19.152906 9184 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.155303 11225 tablet_copy_client.cc:523] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:19.156415 11296 tablet_copy_client.cc:523] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:19.156896 11225 tablet_bootstrap.cc:492] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap starting.
I20240501 08:00:19.157702 11296 tablet_bootstrap.cc:492] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap starting.
I20240501 08:00:19.158144 7558 tablet_service.cc:1514] Processing DeleteTablet for tablet dd17760cd5c2454db97610b2f71c0ee2 with delete_type TABLET_DATA_TOMBSTONED (TS 31b02f06be8a4455b94b1254d77005bd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33196
I20240501 08:00:19.158365 11086 tablet_replica.cc:330] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: stopping tablet replica
I20240501 08:00:19.158531 11086 raft_consensus.cc:2229] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.158677 11086 pending_rounds.cc:62] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: Trying to abort 1 pending ops.
I20240501 08:00:19.158712 11086 pending_rounds.cc:68] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022798516015255552 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "dd17760cd5c2454db97610b2f71c0ee2" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } } }
I20240501 08:00:19.158854 11086 raft_consensus.cc:2894] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 08:00:19.158926 11086 raft_consensus.cc:2866] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 08:00:19.158977 11086 raft_consensus.cc:2258] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.159438 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:19.159823 11086 ts_tablet_manager.cc:1875] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.161917 11086 ts_tablet_manager.cc:1888] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:19.162017 11086 log.cc:1196] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/dd17760cd5c2454db97610b2f71c0ee2
I20240501 08:00:19.162492 7159 catalog_manager.cc:4830] TS 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861): tablet dd17760cd5c2454db97610b2f71c0ee2 (table rebalance_test_table_5 [id=2427aea1b9a6436ca7bd1796a85634ff]) successfully deleted
I20240501 08:00:19.162757 10264 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:19.163924 7447 consensus_queue.cc:237] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:19.165695 7314 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.166033 10264 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:19.166150 10147 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.166842 9449 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 08:00:19.167941 7383 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffc23756b44d4d588e0a8175075d9bc6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.168939 11074 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.169493 11074 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.172528 7314 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.173247 7705 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.173306 7173 catalog_manager.cc:5484] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 4 to 5, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:19.173686 10147 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.174988 7705 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.177132 9450 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:19.180297 7576 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:19.180789 10558 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:19.183641 10558 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.185196 9450 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.187722 7173 catalog_manager.cc:5484] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e reported cstate change: config changed from index 10 to 11, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.190481 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bde42e485c4b49ce8acd694a25ba683a with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.190737 7297 tablet_service.cc:1514] Processing DeleteTablet for tablet bde42e485c4b49ce8acd694a25ba683a with delete_type TABLET_DATA_TOMBSTONED (TS b4f8020266b5455b872a2973fc396578 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:35594
I20240501 08:00:19.190969 10923 tablet_replica.cc:330] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: stopping tablet replica
I20240501 08:00:19.191107 10923 raft_consensus.cc:2229] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.191336 10923 raft_consensus.cc:2258] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.194581 9319 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.198927 10923 ts_tablet_manager.cc:1875] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.199021 9184 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:19.200009 11199 ts_tablet_manager.cc:1888] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:19.200083 11199 log.cc:1196] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b8fdc2b9c2ef44b0bfec59a1a2f6b626
I20240501 08:00:19.200564 11199 tablet_replica.cc:330] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: stopping tablet replica
I20240501 08:00:19.200697 11199 raft_consensus.cc:2229] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.200891 11199 raft_consensus.cc:2258] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.201665 10923 ts_tablet_manager.cc:1888] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 08:00:19.201731 10923 log.cc:1196] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/bde42e485c4b49ce8acd694a25ba683a
I20240501 08:00:19.202225 7159 catalog_manager.cc:4830] TS d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707): tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626 (table rebalance_test_table_0 [id=61deeb642b0b4f06a2ef4b1a391fdcb9]) successfully deleted
I20240501 08:00:19.202220 7162 catalog_manager.cc:4830] TS b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): tablet bde42e485c4b49ce8acd694a25ba683a (table rebalance_test_table_1 [id=ba8a985906c04117a440386d3011b936]) successfully deleted
I20240501 08:00:19.205083 11207 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:19.206743 11199 ts_tablet_manager.cc:1875] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.207847 9184 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.208981 11199 ts_tablet_manager.cc:1888] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 08:00:19.209059 11199 log.cc:1196] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/bde42e485c4b49ce8acd694a25ba683a
I20240501 08:00:19.209610 7159 catalog_manager.cc:4830] TS d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707): tablet bde42e485c4b49ce8acd694a25ba683a (table rebalance_test_table_1 [id=ba8a985906c04117a440386d3011b936]) successfully deleted
I20240501 08:00:19.213742 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:19.217522 10264 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:19.218289 10974 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.220983 9184 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.221091 11040 raft_consensus.cc:2932] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.222970 7173 catalog_manager.cc:5484] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:19.226035 11074 raft_consensus.cc:1050] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f: attempting to promote NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e to VOTER
I20240501 08:00:19.226346 7707 raft_consensus.cc:2932] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.226495 11074 consensus_queue.cc:237] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:19.229812 9449 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.230635 7576 consensus_queue.cc:237] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
W20240501 08:00:19.231456 7383 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffc23756b44d4d588e0a8175075d9bc6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.232074 7314 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.232451 10383 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.232656 9318 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:00:19.233027 11207 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:00:19.234068 9450 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.234462 10383 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.234694 11207 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.236042 7707 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.236485 10383 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.237207 9318 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.239528 7170 catalog_manager.cc:5484] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 reported cstate change: config changed from index 11 to 12, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.239642 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bde42e485c4b49ce8acd694a25ba683a with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.240399 9318 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.242947 9318 raft_consensus.cc:2932] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.244652 7447 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.245718 7170 catalog_manager.cc:5484] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776 reported cstate change: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.246785 10499 ts_tablet_manager.cc:902] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:19.247289 10499 tablet_copy_client.cc:308] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:19.247874 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet e178f7c3c17c4e1ca3d16b6441f76e77 from peer 1c8cb9b1059a47cc822ca13a2b931776 ({username='slave'} at 127.0.99.135:39313)
I20240501 08:00:19.247977 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet e178f7c3c17c4e1ca3d16b6441f76e77 from peer 1c8cb9b1059a47cc822ca13a2b931776 at {username='slave'} at 127.0.99.135:39313: session id = 1c8cb9b1059a47cc822ca13a2b931776-e178f7c3c17c4e1ca3d16b6441f76e77
I20240501 08:00:19.248310 10383 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.248929 7336 tablet_copy_source_session.cc:205] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:19.249819 10499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e178f7c3c17c4e1ca3d16b6441f76e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:19.250813 7707 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.252108 10499 tablet_copy_client.cc:783] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:19.252267 10499 tablet_copy_client.cc:647] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:19.252509 7313 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.256866 7313 consensus_queue.cc:237] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:19.259145 7314 consensus_queue.cc:237] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:19.261998 9319 raft_consensus.cc:1261] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.262190 9318 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:19.262707 7447 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:19.262707 10264 consensus_queue.cc:1035] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.264861 10264 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:19.265385 10264 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:19.266299 11142 raft_consensus.cc:2932] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.267351 9184 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:19.267843 11142 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:19.268009 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e42ef8df4884bc9bc33f5ed4c164c8c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.270079 9450 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:00:19.271163 9318 raft_consensus.cc:2932] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.271333 7576 raft_consensus.cc:858] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
W20240501 08:00:19.271394 7576 raft_consensus.cc:1565] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 8' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:19.271664 7576 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.273905 11142 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.274039 7173 catalog_manager.cc:5484] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 1c8cb9b1059a47cc822ca13a2b931776 reported cstate change: config changed from index 5 to 6, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.274085 7576 raft_consensus.cc:2932] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.275476 7576 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:19.275768 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1088a116ccec4020aec562a69791a188 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.275882 7576 raft_consensus.cc:1261] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.276368 9184 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.276798 7688 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e42ef8df4884bc9bc33f5ed4c164c8c with delete_type TABLET_DATA_TOMBSTONED (TS 09a7b6e2731a474b80411efb62323604 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49464
I20240501 08:00:19.277045 11058 tablet_replica.cc:330] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604: stopping tablet replica
I20240501 08:00:19.277222 11058 raft_consensus.cc:2229] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.277391 11058 raft_consensus.cc:2258] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.278650 11058 ts_tablet_manager.cc:1875] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.279349 11266 tablet_bootstrap.cc:492] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:19.279808 7173 catalog_manager.cc:5484] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 reported cstate change: config changed from index 8 to 9, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.280838 11058 ts_tablet_manager.cc:1888] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:19.280902 11058 log.cc:1196] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 09a7b6e2731a474b80411efb62323604: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/0e42ef8df4884bc9bc33f5ed4c164c8c
I20240501 08:00:19.282220 7162 catalog_manager.cc:4830] TS 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789): tablet 0e42ef8df4884bc9bc33f5ed4c164c8c (table rebalance_test_table_3 [id=013d2bd412784602ac81aa27c405ec0d]) successfully deleted
I20240501 08:00:19.282810 9318 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.283052 11204 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:00:19.283777 11266 tablet_bootstrap.cc:492] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75: Bootstrap complete.
I20240501 08:00:19.283783 9430 tablet_service.cc:1514] Processing DeleteTablet for tablet 1088a116ccec4020aec562a69791a188 with delete_type TABLET_DATA_TOMBSTONED (TS 1807c0c2e7d04b20b0cda311c42eef8e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58736
I20240501 08:00:19.284020 11266 ts_tablet_manager.cc:1367] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75: Time spent bootstrapping tablet: real 0.209s user 0.029s sys 0.000s
I20240501 08:00:19.284466 11266 raft_consensus.cc:348] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:19.284549 11206 raft_consensus.cc:1050] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e to VOTER
I20240501 08:00:19.284583 11266 raft_consensus.cc:729] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b69c5d799e14e438df760453438aa75, State: Initialized, Role: LEARNER
I20240501 08:00:19.284677 11266 consensus_queue.cc:260] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:19.284963 11266 ts_tablet_manager.cc:1398] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:19.285689 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session 3b69c5d799e14e438df760453438aa75-4ea0db42a75a4879b3d3d608666dc2bf received from {username='slave'} at 127.0.99.134:52489
I20240501 08:00:19.285759 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session 3b69c5d799e14e438df760453438aa75-4ea0db42a75a4879b3d3d608666dc2bf on tablet 4ea0db42a75a4879b3d3d608666dc2bf with peer 3b69c5d799e14e438df760453438aa75
I20240501 08:00:19.287148 11206 consensus_queue.cc:237] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:19.287508 7447 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.288532 7448 raft_consensus.cc:1261] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.289000 11203 consensus_queue.cc:1035] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.289095 9450 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.289769 10264 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:19.290428 7170 catalog_manager.cc:5484] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.292038 7709 consensus_queue.cc:237] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:19.292135 7705 consensus_queue.cc:237] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:19.293535 9319 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:19.293753 7448 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:19.293901 11088 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:19.294121 11109 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:19.295140 9449 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.291446 7314 raft_consensus.cc:1261] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.292377 10264 consensus_queue.cc:1035] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20240501 08:00:19.295974 11109 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.296175 7313 raft_consensus.cc:1261] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.296512 11203 consensus_queue.cc:1035] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.297577 7707 consensus_queue.cc:237] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:19.298465 9318 raft_consensus.cc:1261] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.298492 11073 consensus_queue.cc:1035] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.299460 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 81ccdd855b584d65860fd7a5777920a3 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.299445 9318 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.299638 11088 consensus_queue.cc:1035] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.300201 7448 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.300976 9187 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.302526 7168 catalog_manager.cc:5484] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 reported cstate change: config changed from index 8 to 9, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.303417 11109 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.304505 10264 raft_consensus.cc:1050] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 to VOTER
I20240501 08:00:19.303501 11108 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.305027 7314 consensus_queue.cc:237] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:19.305480 9319 raft_consensus.cc:1203] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:00:19.304828 10264 consensus_queue.cc:237] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:19.306270 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df6dd3d8482d4931b1d3014a67e739c6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.306484 9319 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.307096 9056 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.307021 11088 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.307530 10974 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.308384 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5a37e631ae4f46a2be1ad5c43db5724b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.308717 10499 tablet_copy_client.cc:523] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:19.308910 9187 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.310161 10499 tablet_bootstrap.cc:492] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap starting.
W20240501 08:00:19.311707 7160 catalog_manager.cc:4797] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): delete failed for tablet 1088a116ccec4020aec562a69791a188 because the tablet was not found. No further retry: Not found: Tablet not found: 1088a116ccec4020aec562a69791a188
I20240501 08:00:19.312005 7173 catalog_manager.cc:5484] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75 reported cstate change: config changed from index 5 to 6, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.312136 9184 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:19.312505 10323 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:19.312777 7707 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.313318 11142 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.315738 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a37e631ae4f46a2be1ad5c43db5724b with delete_type TABLET_DATA_TOMBSTONED (TS d6b4be7bbd1649d0b7c5e8efe81f759f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45222
I20240501 08:00:19.315976 11199 tablet_replica.cc:330] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: stopping tablet replica
I20240501 08:00:19.316152 11199 raft_consensus.cc:2229] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.316347 9187 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:19.317071 9188 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.317660 9319 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:19.317670 7447 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:19.317919 9450 raft_consensus.cc:1261] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.318046 10264 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:19.318583 10264 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:19.318832 11199 raft_consensus.cc:2258] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.319911 11199 ts_tablet_manager.cc:1875] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.320281 7576 raft_consensus.cc:858] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
W20240501 08:00:19.320346 7576 raft_consensus.cc:1565] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:19.320756 7576 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.321424 11296 tablet_bootstrap.cc:492] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:19.321724 11296 tablet_bootstrap.cc:492] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap complete.
I20240501 08:00:19.321882 11296 ts_tablet_manager.cc:1367] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent bootstrapping tablet: real 0.164s user 0.031s sys 0.000s
I20240501 08:00:19.323153 10974 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.323704 11199 ts_tablet_manager.cc:1888] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:19.323783 11199 log.cc:1196] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5a37e631ae4f46a2be1ad5c43db5724b
I20240501 08:00:19.324283 7159 catalog_manager.cc:4830] TS d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707): tablet 5a37e631ae4f46a2be1ad5c43db5724b (table rebalance_test_table_3 [id=013d2bd412784602ac81aa27c405ec0d]) successfully deleted
I20240501 08:00:19.323719 7447 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.324545 10974 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:19.325932 9319 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.326596 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1088a116ccec4020aec562a69791a188 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.328277 9188 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.329327 9430 tablet_service.cc:1514] Processing DeleteTablet for tablet 81ccdd855b584d65860fd7a5777920a3 with delete_type TABLET_DATA_TOMBSTONED (TS 1807c0c2e7d04b20b0cda311c42eef8e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58736
I20240501 08:00:19.329840 7447 consensus_queue.cc:237] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:19.329828 7168 catalog_manager.cc:5484] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 9 to 10, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.331584 7709 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 08:00:19.332265 7160 catalog_manager.cc:4810] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): delete failed for tablet 81ccdd855b584d65860fd7a5777920a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81ccdd855b584d65860fd7a5777920a3 already in progress: copying tablet
I20240501 08:00:19.332437 11074 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:19.333107 7174 catalog_manager.cc:5484] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 5 to 6, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:19.335402 11074 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.335635 7576 raft_consensus.cc:2932] T 0e42ef8df4884bc9bc33f5ed4c164c8c P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.337136 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ffc23756b44d4d588e0a8175075d9bc6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.338054 7575 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.338603 7709 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.340114 11203 consensus_queue.cc:1035] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.340827 7575 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:19.341202 7575 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 08:00:19.341358 7383 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffc23756b44d4d588e0a8175075d9bc6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.342625 11072 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:19.342772 7558 tablet_service.cc:1514] Processing DeleteTablet for tablet df6dd3d8482d4931b1d3014a67e739c6 with delete_type TABLET_DATA_TOMBSTONED (TS 31b02f06be8a4455b94b1254d77005bd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33196
I20240501 08:00:19.342937 11086 tablet_replica.cc:330] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd: stopping tablet replica
I20240501 08:00:19.343043 11086 raft_consensus.cc:2229] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.343078 9450 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:19.343179 11086 raft_consensus.cc:2258] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.343318 11072 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.343602 10381 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:19.344039 11086 ts_tablet_manager.cc:1875] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.344485 7575 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.344527 9447 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.347127 7575 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:19.347721 7174 catalog_manager.cc:5484] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 6 to 7, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.348667 11086 ts_tablet_manager.cc:1888] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:19.348737 11086 log.cc:1196] T df6dd3d8482d4931b1d3014a67e739c6 P 31b02f06be8a4455b94b1254d77005bd: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/df6dd3d8482d4931b1d3014a67e739c6
I20240501 08:00:19.349139 7316 consensus_queue.cc:237] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:19.349457 11296 raft_consensus.cc:348] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:19.349568 11296 raft_consensus.cc:729] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1807c0c2e7d04b20b0cda311c42eef8e, State: Initialized, Role: LEARNER
I20240501 08:00:19.349696 7159 catalog_manager.cc:4830] TS 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861): tablet df6dd3d8482d4931b1d3014a67e739c6 (table rebalance_test_table_7 [id=c7ddae9ad2184cb9bd097d68171c3fc4]) successfully deleted
I20240501 08:00:19.349666 11296 consensus_queue.cc:260] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:19.351244 7558 tablet_service.cc:1514] Processing DeleteTablet for tablet 1088a116ccec4020aec562a69791a188 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 10)) from {username='slave'} at 127.0.0.1:33196
I20240501 08:00:19.351470 11086 tablet_replica.cc:330] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd: stopping tablet replica
I20240501 08:00:19.351583 11086 raft_consensus.cc:2229] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.351755 11086 raft_consensus.cc:2258] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.352370 11086 ts_tablet_manager.cc:1875] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.353855 7297 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc23756b44d4d588e0a8175075d9bc6 with delete_type TABLET_DATA_TOMBSTONED (TS b4f8020266b5455b872a2973fc396578 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35594
I20240501 08:00:19.354092 10923 tablet_replica.cc:330] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578: stopping tablet replica
I20240501 08:00:19.354245 10923 raft_consensus.cc:2229] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.354419 10923 raft_consensus.cc:2258] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.355167 10923 ts_tablet_manager.cc:1875] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.355433 11086 ts_tablet_manager.cc:1888] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:00:19.355506 11086 log.cc:1196] T 1088a116ccec4020aec562a69791a188 P 31b02f06be8a4455b94b1254d77005bd: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1088a116ccec4020aec562a69791a188
I20240501 08:00:19.355552 11296 ts_tablet_manager.cc:1398] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent starting tablet: real 0.034s user 0.003s sys 0.000s
I20240501 08:00:19.355973 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-81ccdd855b584d65860fd7a5777920a3 received from {username='slave'} at 127.0.99.136:56099
I20240501 08:00:19.356042 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-81ccdd855b584d65860fd7a5777920a3 on tablet 81ccdd855b584d65860fd7a5777920a3 with peer 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:19.356449 7159 catalog_manager.cc:4830] TS 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861): tablet 1088a116ccec4020aec562a69791a188 (table rebalance_test_table_3 [id=013d2bd412784602ac81aa27c405ec0d]) successfully deleted
I20240501 08:00:19.357307 10923 ts_tablet_manager.cc:1888] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:00:19.357365 10923 log.cc:1196] T ffc23756b44d4d588e0a8175075d9bc6 P b4f8020266b5455b872a2973fc396578: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ffc23756b44d4d588e0a8175075d9bc6
I20240501 08:00:19.357831 7162 catalog_manager.cc:4830] TS b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): tablet ffc23756b44d4d588e0a8175075d9bc6 (table rebalance_test_table_8 [id=82bf711832394b1abda2824bb95c9358]) successfully deleted
I20240501 08:00:19.360136 9430 tablet_service.cc:1514] Processing DeleteTablet for tablet 81ccdd855b584d65860fd7a5777920a3 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 9)) from {username='slave'} at 127.0.0.1:58736
I20240501 08:00:19.360986 9319 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:19.361040 11339 tablet_replica.cc:330] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e: stopping tablet replica
I20240501 08:00:19.361274 11339 raft_consensus.cc:2229] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Raft consensus shutting down.
I20240501 08:00:19.361873 7447 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:19.362471 7316 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.362828 10264 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.363533 10264 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:19.363890 11339 raft_consensus.cc:2258] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Raft consensus is shut down!
I20240501 08:00:19.364656 11339 ts_tablet_manager.cc:1875] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.367069 11339 ts_tablet_manager.cc:1888] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:00:19.367225 11339 log.cc:1196] T 81ccdd855b584d65860fd7a5777920a3 P 1807c0c2e7d04b20b0cda311c42eef8e: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/81ccdd855b584d65860fd7a5777920a3
I20240501 08:00:19.367249 10912 tablet_bootstrap.cc:492] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:19.367511 10264 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:19.367592 10912 tablet_bootstrap.cc:492] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap complete.
I20240501 08:00:19.367741 10912 ts_tablet_manager.cc:1367] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent bootstrapping tablet: real 0.256s user 0.028s sys 0.000s
I20240501 08:00:19.367918 7160 catalog_manager.cc:4830] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): tablet 81ccdd855b584d65860fd7a5777920a3 (table rebalance_test_table_5 [id=2427aea1b9a6436ca7bd1796a85634ff]) successfully deleted
I20240501 08:00:19.368175 10912 raft_consensus.cc:348] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:19.368284 10912 raft_consensus.cc:729] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1807c0c2e7d04b20b0cda311c42eef8e, State: Initialized, Role: LEARNER
I20240501 08:00:19.368383 10912 consensus_queue.cc:260] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:19.369104 10912 ts_tablet_manager.cc:1398] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:19.369518 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-4ea0db42a75a4879b3d3d608666dc2bf received from {username='slave'} at 127.0.99.136:56099
I20240501 08:00:19.369590 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-4ea0db42a75a4879b3d3d608666dc2bf on tablet 4ea0db42a75a4879b3d3d608666dc2bf with peer 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:19.371481 9528 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.373661 9319 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.373926 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1088a116ccec4020aec562a69791a188 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.374331 10306 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:19.376760 7168 catalog_manager.cc:5484] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 10 to 11, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:19.376891 7707 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:19.376968 7447 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.377147 9056 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:19.380659 7169 catalog_manager.cc:5484] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 4 to 5, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:19.382443 10558 raft_consensus.cc:2932] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.385025 7313 raft_consensus.cc:2932] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.387131 7447 raft_consensus.cc:2932] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.387495 11225 tablet_bootstrap.cc:492] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:19.387779 11225 tablet_bootstrap.cc:492] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap complete.
I20240501 08:00:19.387974 11225 ts_tablet_manager.cc:1367] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent bootstrapping tablet: real 0.231s user 0.031s sys 0.000s
I20240501 08:00:19.388435 11225 raft_consensus.cc:348] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:19.388564 11225 raft_consensus.cc:729] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8aa5ac376a54e1f8417e5af11d28fa7, State: Initialized, Role: LEARNER
I20240501 08:00:19.388680 11225 consensus_queue.cc:260] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:19.388981 11225 ts_tablet_manager.cc:1398] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:19.389420 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-3d8ae3c82c9d4c5c858fa3f9d5959eb7 received from {username='slave'} at 127.0.99.133:50977
I20240501 08:00:19.389500 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-3d8ae3c82c9d4c5c858fa3f9d5959eb7 on tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7 with peer d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:19.395011 9168 tablet_service.cc:1514] Processing DeleteTablet for tablet bde42e485c4b49ce8acd694a25ba683a with delete_type TABLET_DATA_TOMBSTONED (TS 3b69c5d799e14e438df760453438aa75 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:57020
I20240501 08:00:19.397083 9168 tablet_service.cc:1514] Processing DeleteTablet for tablet 1088a116ccec4020aec562a69791a188 with delete_type TABLET_DATA_TOMBSTONED (TS 3b69c5d799e14e438df760453438aa75 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:57020
I20240501 08:00:19.398072 7173 catalog_manager.cc:5484] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 4 to 5, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.402345 9449 raft_consensus.cc:2932] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.402734 7576 consensus_queue.cc:237] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:19.404072 7447 raft_consensus.cc:1261] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:19.404538 11116 consensus_queue.cc:1035] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:19.405405 7313 raft_consensus.cc:1261] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:19.405705 11372 tablet_replica.cc:330] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75: stopping tablet replica
I20240501 08:00:19.405911 11154 consensus_queue.cc:1035] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:00:19.406396 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4297f95243ac431099a8279c3d9e4b25. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.407624 11372 raft_consensus.cc:2229] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.408062 11372 raft_consensus.cc:2258] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.409466 7576 consensus_queue.cc:237] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:19.410719 9056 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:19.410723 11372 ts_tablet_manager.cc:1875] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.411305 11154 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:19.412895 11116 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:19.413332 9057 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:19.414011 11372 ts_tablet_manager.cc:1888] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240501 08:00:19.414090 11372 log.cc:1196] T bde42e485c4b49ce8acd694a25ba683a P 3b69c5d799e14e438df760453438aa75: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/bde42e485c4b49ce8acd694a25ba683a
I20240501 08:00:19.414723 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cc6f7dcc930e4937865d37cb3c3bb3d2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.415832 7169 catalog_manager.cc:5484] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 reported cstate change: config changed from index 8 to 9, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:19.415942 11204 raft_consensus.cc:2932] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:19.416517 7448 raft_consensus.cc:2932] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:19.418172 7313 raft_consensus.cc:2932] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:19.420225 7170 catalog_manager.cc:5484] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:19.424008 7297 tablet_service.cc:1514] Processing DeleteTablet for tablet cc6f7dcc930e4937865d37cb3c3bb3d2 with delete_type TABLET_DATA_TOMBSTONED (TS b4f8020266b5455b872a2973fc396578 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35594
I20240501 08:00:19.424301 10923 tablet_replica.cc:330] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578: stopping tablet replica
I20240501 08:00:19.424444 10923 raft_consensus.cc:2229] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.424624 10923 raft_consensus.cc:2258] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.425455 10923 ts_tablet_manager.cc:1875] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.428085 10923 ts_tablet_manager.cc:1888] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:00:19.428172 10923 log.cc:1196] T cc6f7dcc930e4937865d37cb3c3bb3d2 P b4f8020266b5455b872a2973fc396578: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/cc6f7dcc930e4937865d37cb3c3bb3d2
I20240501 08:00:19.429092 7162 catalog_manager.cc:4830] TS b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): tablet cc6f7dcc930e4937865d37cb3c3bb3d2 (table rebalance_test_table_2 [id=d578050a40e0474484896c98e4da9f5a]) successfully deleted
I20240501 08:00:19.431561 9449 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:19.432771 11154 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:19.439203 7575 consensus_queue.cc:237] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:19.440050 11372 tablet_replica.cc:330] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75: stopping tablet replica
I20240501 08:00:19.440243 11372 raft_consensus.cc:2229] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.440477 11372 raft_consensus.cc:2258] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.440722 7160 catalog_manager.cc:4830] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): tablet bde42e485c4b49ce8acd694a25ba683a (table rebalance_test_table_1 [id=ba8a985906c04117a440386d3011b936]) successfully deleted
I20240501 08:00:19.441066 7313 raft_consensus.cc:1261] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.441454 11154 consensus_queue.cc:1035] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.441954 11372 ts_tablet_manager.cc:1875] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.443989 7576 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:19.444864 11154 raft_consensus.cc:2932] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.445909 11372 ts_tablet_manager.cc:1888] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 08:00:19.446002 11372 log.cc:1196] T 1088a116ccec4020aec562a69791a188 P 3b69c5d799e14e438df760453438aa75: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/1088a116ccec4020aec562a69791a188
I20240501 08:00:19.446372 9188 raft_consensus.cc:1203] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:00:19.446870 7313 raft_consensus.cc:2932] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.448760 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cd07a00b6817400f86bf8c32bd592d8b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.449077 7160 catalog_manager.cc:4830] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): tablet 1088a116ccec4020aec562a69791a188 (table rebalance_test_table_3 [id=013d2bd412784602ac81aa27c405ec0d]) successfully deleted
I20240501 08:00:19.449470 7166 catalog_manager.cc:5484] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 5 to 6, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.456655 7313 consensus_queue.cc:237] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
W20240501 08:00:19.458590 7251 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5970772c02d04e0b95a2f2feb7dcd655. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.458887 9319 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:19.459323 7448 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:19.459753 10264 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:19.460095 10264 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:19.461208 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet cd07a00b6817400f86bf8c32bd592d8b with delete_type TABLET_DATA_TOMBSTONED (TS d6b4be7bbd1649d0b7c5e8efe81f759f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45222
I20240501 08:00:19.461439 11199 tablet_replica.cc:330] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f: stopping tablet replica
I20240501 08:00:19.461594 11199 raft_consensus.cc:2229] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.461763 11199 raft_consensus.cc:2258] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.462162 11142 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:19.462600 11199 ts_tablet_manager.cc:1875] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.462793 9447 raft_consensus.cc:1261] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.463522 9449 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:19.464774 11199 ts_tablet_manager.cc:1888] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:19.464839 11199 log.cc:1196] T cd07a00b6817400f86bf8c32bd592d8b P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/cd07a00b6817400f86bf8c32bd592d8b
I20240501 08:00:19.465303 11206 consensus_queue.cc:1035] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.466132 7170 catalog_manager.cc:5484] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 6 to 7, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:19.466874 9319 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:19.467684 9449 raft_consensus.cc:2932] T cd07a00b6817400f86bf8c32bd592d8b P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:19.467866 7159 catalog_manager.cc:4830] TS d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707): tablet cd07a00b6817400f86bf8c32bd592d8b (table rebalance_test_table_8 [id=82bf711832394b1abda2824bb95c9358]) successfully deleted
I20240501 08:00:19.471089 7448 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:19.499641 7448 consensus_queue.cc:237] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:19.502413 7313 raft_consensus.cc:1261] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:19.503021 10381 consensus_queue.cc:1035] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:00:19.503942 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8882528b26974c7aa73bb0ff0c444836. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.506160 9057 raft_consensus.cc:1203] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:00:19.511186 7575 raft_consensus.cc:1261] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:19.512459 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:19.513523 10381 consensus_queue.cc:1035] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:19.513816 7576 consensus_queue.cc:237] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:19.513893 10381 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.515127 9319 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:19.515605 11116 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:19.516871 7313 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.518721 7709 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:19.519609 9188 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:19.520642 7576 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.522434 7172 catalog_manager.cc:5484] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index -1 to 4, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:19.526726 10499 tablet_bootstrap.cc:492] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:19.527042 10499 tablet_bootstrap.cc:492] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap complete.
I20240501 08:00:19.527230 10499 ts_tablet_manager.cc:1367] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent bootstrapping tablet: real 0.217s user 0.025s sys 0.004s
I20240501 08:00:19.527688 10499 raft_consensus.cc:348] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:19.527817 10499 raft_consensus.cc:729] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c8cb9b1059a47cc822ca13a2b931776, State: Initialized, Role: LEARNER
I20240501 08:00:19.527923 10499 consensus_queue.cc:260] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:19.528710 11154 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:19.529126 11154 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:19.531381 11154 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.531984 10499 ts_tablet_manager.cc:1398] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 08:00:19.533147 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-e178f7c3c17c4e1ca3d16b6441f76e77 received from {username='slave'} at 127.0.99.135:39313
I20240501 08:00:19.533237 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-e178f7c3c17c4e1ca3d16b6441f76e77 on tablet e178f7c3c17c4e1ca3d16b6441f76e77 with peer 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:19.533397 9319 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.533614 7707 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.536106 7173 catalog_manager.cc:5484] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 reported cstate change: config changed from index 10 to 11, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.536206 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21d76484f5dd499fb6fc3d534bffb779 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.538996 9430 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d76484f5dd499fb6fc3d534bffb779 with delete_type TABLET_DATA_TOMBSTONED (TS 1807c0c2e7d04b20b0cda311c42eef8e not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:58736
I20240501 08:00:19.539275 11339 tablet_replica.cc:330] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e: stopping tablet replica
I20240501 08:00:19.539422 11339 raft_consensus.cc:2229] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.539602 11339 raft_consensus.cc:2258] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.540462 11339 ts_tablet_manager.cc:1875] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.542062 9188 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.542707 11339 ts_tablet_manager.cc:1888] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 08:00:19.542788 11339 log.cc:1196] T 21d76484f5dd499fb6fc3d534bffb779 P 1807c0c2e7d04b20b0cda311c42eef8e: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/21d76484f5dd499fb6fc3d534bffb779
I20240501 08:00:19.543339 7160 catalog_manager.cc:4830] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): tablet 21d76484f5dd499fb6fc3d534bffb779 (table rebalance_test_table_1 [id=ba8a985906c04117a440386d3011b936]) successfully deleted
I20240501 08:00:19.548267 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:19.559051 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:19.560837 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:19.566288 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:19.567945 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:19.572028 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:19.575131 9449 raft_consensus.cc:1203] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:00:19.597093 9184 raft_consensus.cc:1203] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:00:19.598161 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:19.615998 10499 ts_tablet_manager.cc:902] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:19.617112 10499 tablet_copy_client.cc:308] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:19.617835 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet 8311d0921e9e417090e441884615525a from peer 1c8cb9b1059a47cc822ca13a2b931776 ({username='slave'} at 127.0.99.135:36909)
I20240501 08:00:19.617940 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet 8311d0921e9e417090e441884615525a from peer 1c8cb9b1059a47cc822ca13a2b931776 at {username='slave'} at 127.0.99.135:36909: session id = 1c8cb9b1059a47cc822ca13a2b931776-8311d0921e9e417090e441884615525a
I20240501 08:00:19.618865 7599 tablet_copy_source_session.cc:205] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:19.619896 10499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8311d0921e9e417090e441884615525a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:19.621716 7313 consensus_queue.cc:237] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:19.623519 9057 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.623941 10306 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.624245 7707 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 08:00:19.625201 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b357c740aca144b7b063845726f419e2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.625516 11143 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.625758 7576 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.626152 10306 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.626945 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:19.627444 10306 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.630671 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:19.631047 9057 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.631696 7707 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.634577 7576 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.634426 7168 catalog_manager.cc:5484] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 5 to 6, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:19.646940 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:19.654196 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:19.661763 9319 raft_consensus.cc:1203] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:00:19.664372 10499 tablet_copy_client.cc:783] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:19.664635 10499 tablet_copy_client.cc:647] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:19.667011 7313 consensus_queue.cc:237] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:19.668895 7576 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:19.669642 11143 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:19.669770 9057 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:19.670466 11143 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:00:19.670814 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b357c740aca144b7b063845726f419e2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.673087 11143 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.674641 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b357c740aca144b7b063845726f419e2 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.675341 7576 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.678934 7170 catalog_manager.cc:5484] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 6 to 7, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.679491 9057 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:19.682931 7688 tablet_service.cc:1514] Processing DeleteTablet for tablet b357c740aca144b7b063845726f419e2 with delete_type TABLET_DATA_TOMBSTONED (TS 09a7b6e2731a474b80411efb62323604 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49464
I20240501 08:00:19.683161 11058 tablet_replica.cc:330] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604: stopping tablet replica
I20240501 08:00:19.683287 11058 raft_consensus.cc:2229] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.683471 11058 raft_consensus.cc:2258] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.684728 11058 ts_tablet_manager.cc:1875] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.686105 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:19.687186 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:19.686975 11058 ts_tablet_manager.cc:1888] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:00:19.689399 11058 log.cc:1196] T b357c740aca144b7b063845726f419e2 P 09a7b6e2731a474b80411efb62323604: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b357c740aca144b7b063845726f419e2
I20240501 08:00:19.690030 7162 catalog_manager.cc:4830] TS 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789): tablet b357c740aca144b7b063845726f419e2 (table rebalance_test_table_0 [id=61deeb642b0b4f06a2ef4b1a391fdcb9]) successfully deleted
I20240501 08:00:19.692250 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:19.694515 10499 tablet_copy_client.cc:523] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:19.696064 10499 tablet_bootstrap.cc:492] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap starting.
I20240501 08:00:19.704260 10383 raft_consensus.cc:1050] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f: attempting to promote NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 to VOTER
I20240501 08:00:19.704684 10383 consensus_queue.cc:237] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:19.706161 7707 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.706916 11049 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.707985 9319 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.708833 7576 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.709228 11049 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.709693 11049 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.709851 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:19.718144 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:19.726217 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:19.739944 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:19.750735 10499 tablet_bootstrap.cc:492] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:19.751129 10499 tablet_bootstrap.cc:492] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap complete.
I20240501 08:00:19.751396 10499 ts_tablet_manager.cc:1367] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776: Time spent bootstrapping tablet: real 0.055s user 0.026s sys 0.000s
I20240501 08:00:19.751914 10499 raft_consensus.cc:348] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:19.752140 10499 raft_consensus.cc:729] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c8cb9b1059a47cc822ca13a2b931776, State: Initialized, Role: LEARNER
I20240501 08:00:19.760250 10499 consensus_queue.cc:260] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:19.762728 10499 ts_tablet_manager.cc:1398] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240501 08:00:19.763296 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-8311d0921e9e417090e441884615525a received from {username='slave'} at 127.0.99.135:36909
I20240501 08:00:19.763386 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-8311d0921e9e417090e441884615525a on tablet 8311d0921e9e417090e441884615525a with peer 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:19.763891 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:19.773896 11049 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.776945 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:19.777607 9319 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.777813 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:19.778942 7707 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.782891 7576 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.783319 7174 catalog_manager.cc:5484] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776 reported cstate change: config changed from index 4 to 5, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.792965 7448 consensus_queue.cc:237] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:19.794342 9319 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.794763 10383 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.797089 7707 consensus_queue.cc:237] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:19.798992 9184 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:19.800400 11072 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:19.801628 9319 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:19.802239 11072 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:19.804157 11072 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.807889 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 81ccdd855b584d65860fd7a5777920a3 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.808408 9184 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.810351 9319 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.812160 7170 catalog_manager.cc:5484] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 reported cstate change: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.816959 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet 81ccdd855b584d65860fd7a5777920a3 with delete_type TABLET_DATA_TOMBSTONED (TS d6b4be7bbd1649d0b7c5e8efe81f759f not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45222
I20240501 08:00:19.817183 11199 tablet_replica.cc:330] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f: stopping tablet replica
I20240501 08:00:19.817324 11199 raft_consensus.cc:2229] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.817528 11199 raft_consensus.cc:2258] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.818249 11199 ts_tablet_manager.cc:1875] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.820480 11199 ts_tablet_manager.cc:1888] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 08:00:19.820550 11199 log.cc:1196] T 81ccdd855b584d65860fd7a5777920a3 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/81ccdd855b584d65860fd7a5777920a3
I20240501 08:00:19.821108 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:19.821149 7159 catalog_manager.cc:4830] TS d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707): tablet 81ccdd855b584d65860fd7a5777920a3 (table rebalance_test_table_5 [id=2427aea1b9a6436ca7bd1796a85634ff]) successfully deleted
I20240501 08:00:19.824512 7576 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:19.825568 10383 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:19.826234 7576 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:19.830312 10323 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:19.831718 7575 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.834061 10381 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.836023 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5d18a2918c75467a9fe59979ce8936cd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.836365 7575 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.836508 9319 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.839154 7174 catalog_manager.cc:5484] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 5 to 6, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.842986 7688 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d18a2918c75467a9fe59979ce8936cd with delete_type TABLET_DATA_TOMBSTONED (TS 09a7b6e2731a474b80411efb62323604 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49464
I20240501 08:00:19.843194 11058 tablet_replica.cc:330] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604: stopping tablet replica
I20240501 08:00:19.843322 11058 raft_consensus.cc:2229] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.843477 11058 raft_consensus.cc:2258] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.844358 11058 ts_tablet_manager.cc:1875] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.846539 11058 ts_tablet_manager.cc:1888] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:19.846604 11058 log.cc:1196] T 5d18a2918c75467a9fe59979ce8936cd P 09a7b6e2731a474b80411efb62323604: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5d18a2918c75467a9fe59979ce8936cd
I20240501 08:00:19.847100 7162 catalog_manager.cc:4830] TS 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789): tablet 5d18a2918c75467a9fe59979ce8936cd (table rebalance_test_table_8 [id=82bf711832394b1abda2824bb95c9358]) successfully deleted
I20240501 08:00:19.850590 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:19.852766 7575 consensus_queue.cc:237] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:00:19.854408 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7377810e08104526b8b8acdf886a3368. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.855405 7448 raft_consensus.cc:1261] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:19.855965 7688 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d18a2918c75467a9fe59979ce8936cd with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 6)) from {username='slave'} at 127.0.0.1:49464
I20240501 08:00:19.856604 11116 consensus_queue.cc:1035] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:19.856751 9449 raft_consensus.cc:1261] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:19.858173 7162 catalog_manager.cc:4830] TS 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789): tablet 5d18a2918c75467a9fe59979ce8936cd (table rebalance_test_table_8 [id=82bf711832394b1abda2824bb95c9358]) successfully deleted
I20240501 08:00:19.858381 11116 consensus_queue.cc:1035] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:19.859716 11116 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:19.863346 7170 catalog_manager.cc:5484] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 6 to 7, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:19.868391 7448 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:19.871601 7707 consensus_queue.cc:237] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:19.873680 9184 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:19.874310 10323 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:19.875087 9318 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:19.876484 10335 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:19.877593 10335 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.878835 9184 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.879644 9318 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.881508 7166 catalog_manager.cc:5484] T 81ccdd855b584d65860fd7a5777920a3 P 3b69c5d799e14e438df760453438aa75 reported cstate change: config changed from index 10 to 11, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:19.883437 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 81ccdd855b584d65860fd7a5777920a3 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:19.884936 7558 tablet_service.cc:1514] Processing DeleteTablet for tablet 81ccdd855b584d65860fd7a5777920a3 with delete_type TABLET_DATA_TOMBSTONED (TS 31b02f06be8a4455b94b1254d77005bd not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:33196
I20240501 08:00:19.886534 9449 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:19.890892 11521 tablet_replica.cc:330] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd: stopping tablet replica
I20240501 08:00:19.891126 11521 raft_consensus.cc:2229] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:19.891614 11521 raft_consensus.cc:2258] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:19.892410 11521 ts_tablet_manager.cc:1875] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:19.895644 11521 ts_tablet_manager.cc:1888] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 08:00:19.896286 11521 log.cc:1196] T 81ccdd855b584d65860fd7a5777920a3 P 31b02f06be8a4455b94b1254d77005bd: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/81ccdd855b584d65860fd7a5777920a3
I20240501 08:00:19.897104 7575 consensus_queue.cc:237] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:19.898731 7707 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:00:19.899080 7313 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:00:19.899248 7513 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28baea6d7d3f442ba1dbfc5b37de90ac. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.899612 11203 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:19.899973 11203 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:00:19.915254 7159 catalog_manager.cc:4830] TS 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861): tablet 81ccdd855b584d65860fd7a5777920a3 (table rebalance_test_table_5 [id=2427aea1b9a6436ca7bd1796a85634ff]) successfully deleted
I20240501 08:00:19.917436 11203 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:19.920326 7313 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:19.920939 7174 catalog_manager.cc:5484] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:19.921326 7707 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:19.936653 7707 consensus_queue.cc:237] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:19.938939 9449 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:19.939296 9057 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:19.939374 10323 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:00:19.939832 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dde9710a5eb4baa881d9dd04affe184. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.940255 11088 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:19.946550 10323 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:19.949811 9449 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:19.953326 7166 catalog_manager.cc:5484] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e reported cstate change: config changed from index 9 to 10, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:19.979593 11072 raft_consensus.cc:1050] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER 3b69c5d799e14e438df760453438aa75 to VOTER
I20240501 08:00:19.980088 11072 consensus_queue.cc:237] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:19.981879 7448 raft_consensus.cc:1261] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.981863 7313 raft_consensus.cc:1261] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.982146 9184 raft_consensus.cc:1261] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:19.982478 11108 consensus_queue.cc:1035] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.983134 11088 consensus_queue.cc:1035] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.983455 11088 consensus_queue.cc:1035] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:19.987053 7316 consensus_queue.cc:237] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: true } }
I20240501 08:00:19.987314 11579 ts_tablet_manager.cc:902] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7: Initiating tablet copy from peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:19.988273 11579 tablet_copy_client.cc:308] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40707
I20240501 08:00:19.988978 7467 tablet_copy_service.cc:140] P d6b4be7bbd1649d0b7c5e8efe81f759f: Received BeginTabletCopySession request for tablet ffc23756b44d4d588e0a8175075d9bc6 from peer d8aa5ac376a54e1f8417e5af11d28fa7 ({username='slave'} at 127.0.99.133:43147)
W20240501 08:00:19.989022 7251 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5970772c02d04e0b95a2f2feb7dcd655. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.989079 7467 tablet_copy_service.cc:161] P d6b4be7bbd1649d0b7c5e8efe81f759f: Beginning new tablet copy session on tablet ffc23756b44d4d588e0a8175075d9bc6 from peer d8aa5ac376a54e1f8417e5af11d28fa7 at {username='slave'} at 127.0.99.133:43147: session id = d8aa5ac376a54e1f8417e5af11d28fa7-ffc23756b44d4d588e0a8175075d9bc6
W20240501 08:00:19.989279 7253 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789): Couldn't send request to peer 09a7b6e2731a474b80411efb62323604. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.990118 7467 tablet_copy_source_session.cc:205] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:19.990453 10306 raft_consensus.cc:1050] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 to VOTER
I20240501 08:00:19.990836 10306 consensus_queue.cc:237] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:19.991008 11579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc23756b44d4d588e0a8175075d9bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:19.992585 7447 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:19.992622 9057 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:19.992810 9188 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:19.993242 10080 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:19.993695 9528 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:19.994144 9056 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:19.994897 10264 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:19.995436 9318 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:19.996053 7446 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:19.997291 9528 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:19.997614 9528 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:20.001031 11579 tablet_copy_client.cc:783] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:20.001235 11579 tablet_copy_client.cc:647] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:20.001499 10264 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: true } } }
I20240501 08:00:20.004678 7172 catalog_manager.cc:5484] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 7 to 8, NON_VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:20.007678 7447 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: true } } }
I20240501 08:00:20.008183 9318 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: true } } }
I20240501 08:00:20.009816 11579 tablet_copy_client.cc:523] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:20.011422 11579 tablet_bootstrap.cc:492] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap starting.
I20240501 08:00:20.012579 10306 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.014331 9188 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.021510 9056 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.022886 7447 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.024470 7168 catalog_manager.cc:5484] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 7 to 8, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:20.030920 9319 raft_consensus.cc:1203] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:00:20.037073 11605 ts_tablet_manager.cc:902] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:20.037760 11605 tablet_copy_client.cc:308] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:20.038628 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet 4297f95243ac431099a8279c3d9e4b25 from peer 3b69c5d799e14e438df760453438aa75 ({username='slave'} at 127.0.99.134:51835)
I20240501 08:00:20.038750 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet 4297f95243ac431099a8279c3d9e4b25 from peer 3b69c5d799e14e438df760453438aa75 at {username='slave'} at 127.0.99.134:51835: session id = 3b69c5d799e14e438df760453438aa75-4297f95243ac431099a8279c3d9e4b25
I20240501 08:00:20.039825 7599 tablet_copy_source_session.cc:205] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:20.040841 11605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4297f95243ac431099a8279c3d9e4b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:20.040896 11088 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.042755 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:20.042932 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:20.043924 11605 tablet_copy_client.cc:783] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:20.044076 11605 tablet_copy_client.cc:647] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:20.044153 9184 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.046097 7448 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.046595 7166 catalog_manager.cc:5484] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75 reported cstate change: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.050597 7707 consensus_queue.cc:237] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:20.051405 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:20.051898 9184 raft_consensus.cc:1261] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:20.051946 7448 raft_consensus.cc:1261] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:20.052330 10323 consensus_queue.cc:1035] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:20.052469 11072 consensus_queue.cc:1035] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:20.054715 10323 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.056197 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 07967f68a412411494c6dd1f47209009 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:20.057018 9184 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.063647 7168 catalog_manager.cc:5484] T 07967f68a412411494c6dd1f47209009 P 3b69c5d799e14e438df760453438aa75 reported cstate change: config changed from index 5 to 6, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.066718 7297 tablet_service.cc:1514] Processing DeleteTablet for tablet 07967f68a412411494c6dd1f47209009 with delete_type TABLET_DATA_TOMBSTONED (TS b4f8020266b5455b872a2973fc396578 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35594
I20240501 08:00:20.071065 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:20.084097 7448 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.085808 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:20.088027 11108 raft_consensus.cc:1050] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER 3b69c5d799e14e438df760453438aa75 to VOTER
I20240501 08:00:20.088474 11108 consensus_queue.cc:237] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:20.090703 9184 raft_consensus.cc:1261] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:20.090723 9449 raft_consensus.cc:1261] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:20.091027 11605 tablet_copy_client.cc:523] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:20.091351 7448 raft_consensus.cc:1261] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:20.091778 11072 consensus_queue.cc:1035] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:20.092547 11109 consensus_queue.cc:1035] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:20.092814 11109 consensus_queue.cc:1035] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:20.092988 11605 tablet_bootstrap.cc:492] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75: Bootstrap starting.
I20240501 08:00:20.094954 7316 raft_consensus.cc:1261] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:20.095530 11088 consensus_queue.cc:1035] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:20.095649 11637 tablet_replica.cc:330] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: stopping tablet replica
I20240501 08:00:20.095775 11637 raft_consensus.cc:2229] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:20.095928 11637 pending_rounds.cc:62] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: Trying to abort 1 pending ops.
I20240501 08:00:20.096002 11637 pending_rounds.cc:68] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022798520237281280 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "07967f68a412411494c6dd1f47209009" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } } }
I20240501 08:00:20.096822 11637 raft_consensus.cc:2894] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 08:00:20.096897 11637 raft_consensus.cc:2866] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 08:00:20.096951 11637 raft_consensus.cc:2258] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:20.099783 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:20.101469 7447 consensus_queue.cc:237] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
W20240501 08:00:20.102846 11597 consensus_queue.cc:1159] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b4f8020266b5455b872a2973fc396578" status { last_received { term: 1 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 5 } }
I20240501 08:00:20.103112 9188 raft_consensus.cc:1261] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:20.103328 7575 raft_consensus.cc:1261] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:20.103590 11637 ts_tablet_manager.cc:1875] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:20.103572 11049 consensus_queue.cc:1035] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:00:20.103799 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f647cfc562794e65a05918b241014d72. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:20.104008 11049 consensus_queue.cc:1035] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:20.106077 11637 ts_tablet_manager.cc:1888] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:20.106212 11637 log.cc:1196] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/07967f68a412411494c6dd1f47209009
I20240501 08:00:20.106925 7162 catalog_manager.cc:4830] TS b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): tablet 07967f68a412411494c6dd1f47209009 (table rebalance_test_table_4 [id=ae84a8bf8ac745fb982412e9b194d7fc]) successfully deleted
I20240501 08:00:20.111404 10381 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.111693 9188 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.114459 7168 catalog_manager.cc:5484] T f647cfc562794e65a05918b241014d72 P 3b69c5d799e14e438df760453438aa75 reported cstate change: config changed from index 6 to 7, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.116048 7575 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.117718 7447 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ffc23756b44d4d588e0a8175075d9bc6"
dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58424
I20240501 08:00:20.117873 7447 raft_consensus.cc:595] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:20.119025 11088 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.121778 7316 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.123944 9184 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.125191 7173 catalog_manager.cc:5484] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 5 to 6, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.125967 7448 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.131791 7707 consensus_queue.cc:237] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:20.133844 9184 raft_consensus.cc:1261] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:20.134303 10323 consensus_queue.cc:1035] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:20.135162 7316 raft_consensus.cc:1261] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:20.135603 10323 consensus_queue.cc:1035] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:20.136263 7448 raft_consensus.cc:1261] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:20.136674 10323 consensus_queue.cc:1035] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:20.144886 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:20.147547 10323 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.149861 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4ea0db42a75a4879b3d3d608666dc2bf with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:20.151844 7172 catalog_manager.cc:5484] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 6 to 7, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:20.152078 9184 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.155287 7316 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.156033 9430 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ea0db42a75a4879b3d3d608666dc2bf with delete_type TABLET_DATA_TOMBSTONED (TS 1807c0c2e7d04b20b0cda311c42eef8e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58736
I20240501 08:00:20.156370 11686 tablet_replica.cc:330] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e: stopping tablet replica
I20240501 08:00:20.156522 11686 raft_consensus.cc:2229] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Raft consensus shutting down.
I20240501 08:00:20.156769 11686 pending_rounds.cc:62] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e: Trying to abort 1 pending ops.
I20240501 08:00:20.156833 11686 pending_rounds.cc:68] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7022798520681201664 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "4ea0db42a75a4879b3d3d608666dc2bf" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } } }
I20240501 08:00:20.157014 11686 raft_consensus.cc:2894] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240501 08:00:20.157109 11686 raft_consensus.cc:2866] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 08:00:20.157207 11686 raft_consensus.cc:2258] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Raft consensus is shut down!
I20240501 08:00:20.157616 7707 consensus_queue.cc:237] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:20.158700 9184 raft_consensus.cc:1261] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:20.159008 7447 raft_consensus.cc:1261] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:20.159173 11685 ts_tablet_manager.cc:902] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:20.159233 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:20.159410 11088 consensus_queue.cc:1035] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:20.159514 7448 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.159776 11685 tablet_copy_client.cc:308] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:20.160728 11605 tablet_bootstrap.cc:492] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:20.161026 11605 tablet_bootstrap.cc:492] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75: Bootstrap complete.
I20240501 08:00:20.161154 11088 consensus_queue.cc:1035] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:20.161216 11605 ts_tablet_manager.cc:1367] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75: Time spent bootstrapping tablet: real 0.068s user 0.025s sys 0.003s
I20240501 08:00:20.161695 11605 raft_consensus.cc:348] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:20.161808 11605 raft_consensus.cc:729] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b69c5d799e14e438df760453438aa75, State: Initialized, Role: LEARNER
I20240501 08:00:20.161893 11605 consensus_queue.cc:260] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:20.169346 11605 ts_tablet_manager.cc:1398] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 08:00:20.169880 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session 3b69c5d799e14e438df760453438aa75-4297f95243ac431099a8279c3d9e4b25 received from {username='slave'} at 127.0.99.134:51835
I20240501 08:00:20.169974 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session 3b69c5d799e14e438df760453438aa75-4297f95243ac431099a8279c3d9e4b25 on tablet 4297f95243ac431099a8279c3d9e4b25 with peer 3b69c5d799e14e438df760453438aa75
I20240501 08:00:20.170014 10323 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.171814 7447 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.173717 11660 ts_tablet_manager.cc:902] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7: Initiating tablet copy from peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:20.174289 11660 tablet_copy_client.cc:308] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40707
I20240501 08:00:20.174319 7170 catalog_manager.cc:5484] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 7 to 8, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.174726 7467 tablet_copy_service.cc:140] P d6b4be7bbd1649d0b7c5e8efe81f759f: Received BeginTabletCopySession request for tablet 8882528b26974c7aa73bb0ff0c444836 from peer d8aa5ac376a54e1f8417e5af11d28fa7 ({username='slave'} at 127.0.99.133:43147)
I20240501 08:00:20.174825 7467 tablet_copy_service.cc:161] P d6b4be7bbd1649d0b7c5e8efe81f759f: Beginning new tablet copy session on tablet 8882528b26974c7aa73bb0ff0c444836 from peer d8aa5ac376a54e1f8417e5af11d28fa7 at {username='slave'} at 127.0.99.133:43147: session id = d8aa5ac376a54e1f8417e5af11d28fa7-8882528b26974c7aa73bb0ff0c444836
I20240501 08:00:20.175762 7467 tablet_copy_source_session.cc:205] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:20.177026 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4ea0db42a75a4879b3d3d608666dc2bf with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:20.176600 11660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8882528b26974c7aa73bb0ff0c444836. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:20.177508 7297 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ea0db42a75a4879b3d3d608666dc2bf with delete_type TABLET_DATA_TOMBSTONED (TS b4f8020266b5455b872a2973fc396578 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35594
I20240501 08:00:20.177744 11637 tablet_replica.cc:330] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578: stopping tablet replica
I20240501 08:00:20.177881 11637 raft_consensus.cc:2229] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:20.178115 11637 raft_consensus.cc:2258] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:20.178951 11637 ts_tablet_manager.cc:1875] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:20.179652 9184 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.184005 11637 ts_tablet_manager.cc:1888] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:00:20.184095 11637 log.cc:1196] T 4ea0db42a75a4879b3d3d608666dc2bf P b4f8020266b5455b872a2973fc396578: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4ea0db42a75a4879b3d3d608666dc2bf
I20240501 08:00:20.184649 7162 catalog_manager.cc:4830] TS b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): tablet 4ea0db42a75a4879b3d3d608666dc2bf (table rebalance_test_table_6 [id=22d40ca8e4714c2495974e28872c5687]) successfully deleted
I20240501 08:00:20.186437 11660 tablet_copy_client.cc:783] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:20.186597 11660 tablet_copy_client.cc:647] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:20.187793 11660 tablet_copy_client.cc:523] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:20.189193 11660 tablet_bootstrap.cc:492] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap starting.
W20240501 08:00:20.204483 11643 consensus_queue.cc:1159] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" status { last_received { term: 1 index: 6 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 6 } }
I20240501 08:00:20.208747 11686 ts_tablet_manager.cc:1875] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:20.211083 11686 ts_tablet_manager.cc:1888] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:00:20.211170 11686 log.cc:1196] T 4ea0db42a75a4879b3d3d608666dc2bf P 1807c0c2e7d04b20b0cda311c42eef8e: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/4ea0db42a75a4879b3d3d608666dc2bf
I20240501 08:00:20.211783 7160 catalog_manager.cc:4830] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): tablet 4ea0db42a75a4879b3d3d608666dc2bf (table rebalance_test_table_6 [id=22d40ca8e4714c2495974e28872c5687]) successfully deleted
I20240501 08:00:20.212109 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet b357c740aca144b7b063845726f419e2 from peer 1807c0c2e7d04b20b0cda311c42eef8e ({username='slave'} at 127.0.99.136:33637)
I20240501 08:00:20.212235 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet b357c740aca144b7b063845726f419e2 from peer 1807c0c2e7d04b20b0cda311c42eef8e at {username='slave'} at 127.0.99.136:33637: session id = 1807c0c2e7d04b20b0cda311c42eef8e-b357c740aca144b7b063845726f419e2
I20240501 08:00:20.213090 7336 tablet_copy_source_session.cc:205] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:20.213908 11685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b357c740aca144b7b063845726f419e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:20.215996 11579 tablet_bootstrap.cc:492] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:20.216050 11685 tablet_copy_client.cc:783] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:20.216198 11685 tablet_copy_client.cc:647] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:20.216302 11579 tablet_bootstrap.cc:492] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap complete.
I20240501 08:00:20.217398 11579 ts_tablet_manager.cc:1367] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent bootstrapping tablet: real 0.206s user 0.025s sys 0.005s
I20240501 08:00:20.217401 11685 tablet_copy_client.cc:523] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:20.217860 11579 raft_consensus.cc:348] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:20.218008 11579 raft_consensus.cc:729] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8aa5ac376a54e1f8417e5af11d28fa7, State: Initialized, Role: LEARNER
I20240501 08:00:20.218119 11579 consensus_queue.cc:260] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:20.218633 11685 tablet_bootstrap.cc:492] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap starting.
I20240501 08:00:20.220757 11579 ts_tablet_manager.cc:1398] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 08:00:20.221267 7467 tablet_copy_service.cc:342] P d6b4be7bbd1649d0b7c5e8efe81f759f: Request end of tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-ffc23756b44d4d588e0a8175075d9bc6 received from {username='slave'} at 127.0.99.133:43147
I20240501 08:00:20.221343 7467 tablet_copy_service.cc:433] P d6b4be7bbd1649d0b7c5e8efe81f759f: ending tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-ffc23756b44d4d588e0a8175075d9bc6 on tablet ffc23756b44d4d588e0a8175075d9bc6 with peer d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:20.246604 7707 consensus_queue.cc:237] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:20.247994 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:20.248894 9319 raft_consensus.cc:1261] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:20.249573 7316 raft_consensus.cc:1261] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:20.249926 11109 consensus_queue.cc:1035] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:20.250099 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:00:20.250866 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df6dd3d8482d4931b1d3014a67e739c6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:20.252517 11109 consensus_queue.cc:1035] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:20.253886 7688 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:20.254503 11109 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:20.254729 10306 raft_consensus.cc:1050] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 to VOTER
I20240501 08:00:20.255072 10306 consensus_queue.cc:237] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:20.256640 7428 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:20.257299 9056 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:20.257826 11143 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:20.259321 7170 catalog_manager.cc:5484] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 6 to 7, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:20.262943 9318 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:20.263513 7316 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:20.266232 7575 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:20.268028 9319 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:20.271095 11040 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:20.271234 10383 raft_consensus.cc:979] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f: : Instructing follower 09a7b6e2731a474b80411efb62323604 to start an election
I20240501 08:00:20.271318 10383 raft_consensus.cc:1067] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Signalling peer 09a7b6e2731a474b80411efb62323604 to start an election
I20240501 08:00:20.271456 11040 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:20.271859 7709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ffc23756b44d4d588e0a8175075d9bc6"
dest_uuid: "09a7b6e2731a474b80411efb62323604"
from {username='slave'} at 127.0.99.130:59719
I20240501 08:00:20.272009 7709 raft_consensus.cc:482] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:20.272063 7709 raft_consensus.cc:3037] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:20.273164 7709 raft_consensus.cc:504] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:20.273600 7709 leader_election.cc:288] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 2 election: Requested vote from peers d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707), 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587)
I20240501 08:00:20.275395 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc23756b44d4d588e0a8175075d9bc6" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f"
I20240501 08:00:20.275415 9449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc23756b44d4d588e0a8175075d9bc6" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "1807c0c2e7d04b20b0cda311c42eef8e"
I20240501 08:00:20.275521 7447 raft_consensus.cc:3032] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:20.275560 7447 raft_consensus.cc:729] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Running, Role: LEADER
I20240501 08:00:20.275584 9449 raft_consensus.cc:3037] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:20.275724 7447 consensus_queue.cc:260] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:20.275892 7447 raft_consensus.cc:3037] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:20.276684 9449 raft_consensus.cc:2454] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 2.
I20240501 08:00:20.276810 7447 raft_consensus.cc:2454] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 2.
I20240501 08:00:20.277206 7643 leader_election.cc:302] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, d6b4be7bbd1649d0b7c5e8efe81f759f; no voters:
I20240501 08:00:20.277415 11108 raft_consensus.cc:2781] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:20.277489 11108 raft_consensus.cc:686] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 2 LEADER]: Becoming Leader. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Running, Role: LEADER
I20240501 08:00:20.277639 11108 consensus_queue.cc:237] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:20.279997 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:20.280887 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:20.285169 7168 catalog_manager.cc:5484] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 reported cstate change: term changed from 1 to 2, leader changed from d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) to 09a7b6e2731a474b80411efb62323604 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:20.289499 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:20.290741 11685 tablet_bootstrap.cc:492] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:20.291132 11685 tablet_bootstrap.cc:492] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap complete.
I20240501 08:00:20.291355 11685 ts_tablet_manager.cc:1367] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent bootstrapping tablet: real 0.073s user 0.028s sys 0.000s
I20240501 08:00:20.292008 11685 raft_consensus.cc:348] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:20.292212 11685 raft_consensus.cc:729] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1807c0c2e7d04b20b0cda311c42eef8e, State: Initialized, Role: LEARNER
I20240501 08:00:20.292387 11685 consensus_queue.cc:260] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:20.293354 11685 ts_tablet_manager.cc:1398] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:20.294548 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-b357c740aca144b7b063845726f419e2 received from {username='slave'} at 127.0.99.136:33637
I20240501 08:00:20.294632 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-b357c740aca144b7b063845726f419e2 on tablet b357c740aca144b7b063845726f419e2 with peer 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:20.304174 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:20.308072 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:20.310478 7688 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:20.311978 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:20.315655 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:20.318930 11143 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:20.321422 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:20.321800 7575 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:20.321952 7168 catalog_manager.cc:5484] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:20.324321 9319 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:20.336679 7316 consensus_queue.cc:237] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:20.336881 9056 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:20.339134 9319 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:20.339277 11660 tablet_bootstrap.cc:492] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:20.339677 11660 tablet_bootstrap.cc:492] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap complete.
I20240501 08:00:20.340351 7428 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:20.340698 10264 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:20.340922 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:20.341677 11660 ts_tablet_manager.cc:1367] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent bootstrapping tablet: real 0.153s user 0.029s sys 0.002s
I20240501 08:00:20.342232 11660 raft_consensus.cc:348] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:20.342423 11660 raft_consensus.cc:729] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8aa5ac376a54e1f8417e5af11d28fa7, State: Initialized, Role: LEARNER
I20240501 08:00:20.342571 11660 consensus_queue.cc:260] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:20.342943 11660 ts_tablet_manager.cc:1398] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:20.343425 7467 tablet_copy_service.cc:342] P d6b4be7bbd1649d0b7c5e8efe81f759f: Request end of tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-8882528b26974c7aa73bb0ff0c444836 received from {username='slave'} at 127.0.99.133:43147
I20240501 08:00:20.343501 7467 tablet_copy_service.cc:433] P d6b4be7bbd1649d0b7c5e8efe81f759f: ending tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-8882528b26974c7aa73bb0ff0c444836 on tablet 8882528b26974c7aa73bb0ff0c444836 with peer d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:20.345103 9056 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:20.351393 10264 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:20.352262 11605 ts_tablet_manager.cc:902] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:20.352478 10264 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:20.352699 11605 tablet_copy_client.cc:308] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:20.353188 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet 28baea6d7d3f442ba1dbfc5b37de90ac from peer 3b69c5d799e14e438df760453438aa75 ({username='slave'} at 127.0.99.134:51835)
I20240501 08:00:20.353308 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet 28baea6d7d3f442ba1dbfc5b37de90ac from peer 3b69c5d799e14e438df760453438aa75 at {username='slave'} at 127.0.99.134:51835: session id = 3b69c5d799e14e438df760453438aa75-28baea6d7d3f442ba1dbfc5b37de90ac
I20240501 08:00:20.354279 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e178f7c3c17c4e1ca3d16b6441f76e77 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:20.354990 9318 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:20.355836 7599 tablet_copy_source_session.cc:205] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:20.356719 11605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28baea6d7d3f442ba1dbfc5b37de90ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:20.357847 7168 catalog_manager.cc:5484] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776 reported cstate change: config changed from index 8 to 9, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:20.358981 11605 tablet_copy_client.cc:783] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:20.359129 11605 tablet_copy_client.cc:647] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:20.363790 9056 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:20.368682 7558 tablet_service.cc:1514] Processing DeleteTablet for tablet e178f7c3c17c4e1ca3d16b6441f76e77 with delete_type TABLET_DATA_TOMBSTONED (TS 31b02f06be8a4455b94b1254d77005bd not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33196
I20240501 08:00:20.368911 11521 tablet_replica.cc:330] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: stopping tablet replica
I20240501 08:00:20.369062 11521 raft_consensus.cc:2229] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:20.369313 11521 raft_consensus.cc:2258] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:20.370081 11521 ts_tablet_manager.cc:1875] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:20.372351 11521 ts_tablet_manager.cc:1888] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:00:20.372421 11521 log.cc:1196] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e178f7c3c17c4e1ca3d16b6441f76e77
I20240501 08:00:20.373365 7159 catalog_manager.cc:4830] TS 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861): tablet e178f7c3c17c4e1ca3d16b6441f76e77 (table rebalance_test_table_5 [id=2427aea1b9a6436ca7bd1796a85634ff]) successfully deleted
I20240501 08:00:20.382663 11745 ts_tablet_manager.cc:902] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:20.383723 11745 tablet_copy_client.cc:308] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:20.384354 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet 5970772c02d04e0b95a2f2feb7dcd655 from peer 09a7b6e2731a474b80411efb62323604 ({username='slave'} at 127.0.99.132:42259)
I20240501 08:00:20.384506 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet 5970772c02d04e0b95a2f2feb7dcd655 from peer 09a7b6e2731a474b80411efb62323604 at {username='slave'} at 127.0.99.132:42259: session id = 09a7b6e2731a474b80411efb62323604-5970772c02d04e0b95a2f2feb7dcd655
I20240501 08:00:20.385448 7336 tablet_copy_source_session.cc:205] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:20.387416 11745 ts_tablet_manager.cc:1875] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:00:20.389724 11745 ts_tablet_manager.cc:1888] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 08:00:20.389912 11745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5970772c02d04e0b95a2f2feb7dcd655. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:20.391355 11745 tablet_copy_client.cc:783] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:20.391574 11745 tablet_copy_client.cc:647] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:20.393051 11745 tablet_copy_client.cc:523] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:20.394852 11745 tablet_bootstrap.cc:492] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
I20240501 08:00:20.424897 11745 tablet_bootstrap.cc:492] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:20.425263 11745 tablet_bootstrap.cc:492] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: Bootstrap complete.
I20240501 08:00:20.425468 11745 ts_tablet_manager.cc:1367] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.007s
I20240501 08:00:20.425907 11745 raft_consensus.cc:348] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: true } }
I20240501 08:00:20.426074 11745 raft_consensus.cc:729] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: LEARNER
I20240501 08:00:20.426195 11745 consensus_queue.cc:260] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: true } }
I20240501 08:00:20.429888 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:20.430748 11745 ts_tablet_manager.cc:1398] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:00:20.432780 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session 09a7b6e2731a474b80411efb62323604-5970772c02d04e0b95a2f2feb7dcd655 received from {username='slave'} at 127.0.99.132:42259
I20240501 08:00:20.432869 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session 09a7b6e2731a474b80411efb62323604-5970772c02d04e0b95a2f2feb7dcd655 on tablet 5970772c02d04e0b95a2f2feb7dcd655 with peer 09a7b6e2731a474b80411efb62323604
I20240501 08:00:20.433789 11605 tablet_copy_client.cc:523] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:20.435325 11605 tablet_bootstrap.cc:492] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75: Bootstrap starting.
I20240501 08:00:20.452771 11769 ts_tablet_manager.cc:902] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:20.453672 11769 tablet_copy_client.cc:308] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:20.454212 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet 7377810e08104526b8b8acdf886a3368 from peer 1c8cb9b1059a47cc822ca13a2b931776 ({username='slave'} at 127.0.99.135:36909)
I20240501 08:00:20.454306 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet 7377810e08104526b8b8acdf886a3368 from peer 1c8cb9b1059a47cc822ca13a2b931776 at {username='slave'} at 127.0.99.135:36909: session id = 1c8cb9b1059a47cc822ca13a2b931776-7377810e08104526b8b8acdf886a3368
I20240501 08:00:20.455217 7599 tablet_copy_source_session.cc:205] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:20.456486 11769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7377810e08104526b8b8acdf886a3368. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:20.460724 9184 raft_consensus.cc:1203] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:00:20.467018 11781 ts_tablet_manager.cc:902] T 5970772c02d04e0b95a2f2feb7dcd655 P 3b69c5d799e14e438df760453438aa75: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:20.467617 11781 tablet_copy_client.cc:308] T 5970772c02d04e0b95a2f2feb7dcd655 P 3b69c5d799e14e438df760453438aa75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:20.468283 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet 5970772c02d04e0b95a2f2feb7dcd655 from peer 3b69c5d799e14e438df760453438aa75 ({username='slave'} at 127.0.99.134:33617)
I20240501 08:00:20.468395 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet 5970772c02d04e0b95a2f2feb7dcd655 from peer 3b69c5d799e14e438df760453438aa75 at {username='slave'} at 127.0.99.134:33617: session id = 3b69c5d799e14e438df760453438aa75-5970772c02d04e0b95a2f2feb7dcd655
I20240501 08:00:20.469422 7336 tablet_copy_source_session.cc:205] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:20.470877 11781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5970772c02d04e0b95a2f2feb7dcd655. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:20.472812 7575 consensus_queue.cc:237] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } }
I20240501 08:00:20.474085 11781 tablet_copy_client.cc:783] T 5970772c02d04e0b95a2f2feb7dcd655 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:20.474323 11781 tablet_copy_client.cc:647] T 5970772c02d04e0b95a2f2feb7dcd655 P 3b69c5d799e14e438df760453438aa75: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:20.474593 7707 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:00:20.474856 9184 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:00:20.474989 10907 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:00:20.475212 9319 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:00:20.475531 11781 tablet_copy_client.cc:523] T 5970772c02d04e0b95a2f2feb7dcd655 P 3b69c5d799e14e438df760453438aa75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:20.476023 11769 tablet_copy_client.cc:783] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:20.476200 11769 tablet_copy_client.cc:647] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:20.477327 11781 tablet_bootstrap.cc:492] T 5970772c02d04e0b95a2f2feb7dcd655 P 3b69c5d799e14e438df760453438aa75: Bootstrap starting.
I20240501 08:00:20.482472 11203 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:00:20.482870 11203 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:00:20.485893 11769 tablet_copy_client.cc:523] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:20.487530 11769 tablet_bootstrap.cc:492] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap starting.
I20240501 08:00:20.490171 11605 tablet_bootstrap.cc:492] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:20.490475 11605 tablet_bootstrap.cc:492] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75: Bootstrap complete.
I20240501 08:00:20.490665 11605 ts_tablet_manager.cc:1367] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75: Time spent bootstrapping tablet: real 0.055s user 0.024s sys 0.003s
I20240501 08:00:20.491122 11605 raft_consensus.cc:348] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:20.491247 11605 raft_consensus.cc:729] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b69c5d799e14e438df760453438aa75, State: Initialized, Role: LEARNER
I20240501 08:00:20.491358 11605 consensus_queue.cc:260] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:20.491669 11605 ts_tablet_manager.cc:1398] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:20.492276 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session 3b69c5d799e14e438df760453438aa75-28baea6d7d3f442ba1dbfc5b37de90ac received from {username='slave'} at 127.0.99.134:51835
I20240501 08:00:20.492372 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session 3b69c5d799e14e438df760453438aa75-28baea6d7d3f442ba1dbfc5b37de90ac on tablet 28baea6d7d3f442ba1dbfc5b37de90ac with peer 3b69c5d799e14e438df760453438aa75
I20240501 08:00:20.495466 10907 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } }
I20240501 08:00:20.496362 9184 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } }
I20240501 08:00:20.496362 9319 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } }
I20240501 08:00:20.499101 7167 catalog_manager.cc:5484] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 11 to 12. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } }
I20240501 08:00:20.502547 7575 consensus_queue.cc:237] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:20.504181 7709 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 08:00:20.504830 9184 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 08:00:20.504891 7707 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } }
I20240501 08:00:20.505251 10558 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:00:20.507124 10558 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:00:20.509224 11154 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.510164 7709 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.510404 9184 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.513478 7167 catalog_manager.cc:5484] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 12 to 13, VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.515532 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21d76484f5dd499fb6fc3d534bffb779 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:20.518565 9298 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d76484f5dd499fb6fc3d534bffb779 with delete_type TABLET_DATA_TOMBSTONED (TS 1c8cb9b1059a47cc822ca13a2b931776 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:40266
I20240501 08:00:20.518947 11685 ts_tablet_manager.cc:902] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e: Initiating tablet copy from peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:20.519377 11685 tablet_copy_client.cc:308] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40707
I20240501 08:00:20.520152 7467 tablet_copy_service.cc:140] P d6b4be7bbd1649d0b7c5e8efe81f759f: Received BeginTabletCopySession request for tablet f647cfc562794e65a05918b241014d72 from peer 1807c0c2e7d04b20b0cda311c42eef8e ({username='slave'} at 127.0.99.136:53469)
I20240501 08:00:20.520262 7467 tablet_copy_service.cc:161] P d6b4be7bbd1649d0b7c5e8efe81f759f: Beginning new tablet copy session on tablet f647cfc562794e65a05918b241014d72 from peer 1807c0c2e7d04b20b0cda311c42eef8e at {username='slave'} at 127.0.99.136:53469: session id = 1807c0c2e7d04b20b0cda311c42eef8e-f647cfc562794e65a05918b241014d72
I20240501 08:00:20.521225 7467 tablet_copy_source_session.cc:205] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:20.521605 11809 tablet_replica.cc:330] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: stopping tablet replica
I20240501 08:00:20.522151 11809 raft_consensus.cc:2229] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:20.522598 11809 raft_consensus.cc:2258] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:20.522740 7575 consensus_queue.cc:237] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:20.523788 11809 ts_tablet_manager.cc:1875] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:20.524875 11685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f647cfc562794e65a05918b241014d72. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:00:20.524981 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:20.525509 7575 consensus_queue.cc:237] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
I20240501 08:00:20.526419 11809 ts_tablet_manager.cc:1888] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240501 08:00:20.526520 11809 log.cc:1196] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/21d76484f5dd499fb6fc3d534bffb779
I20240501 08:00:20.527037 9319 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 08:00:20.527772 9184 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 08:00:20.527884 10558 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:00:20.528431 9449 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 08:00:20.528533 7709 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 08:00:20.528874 11203 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:00:20.529017 10558 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 08:00:20.529356 7515 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): Couldn't send request to peer b4f8020266b5455b872a2973fc396578. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:20.530072 11685 tablet_copy_client.cc:783] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:20.530226 11685 tablet_copy_client.cc:647] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:20.530287 7159 catalog_manager.cc:4830] TS 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): tablet 21d76484f5dd499fb6fc3d534bffb779 (table rebalance_test_table_1 [id=ba8a985906c04117a440386d3011b936]) successfully deleted
I20240501 08:00:20.530273 10558 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:00:20.532042 10558 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:20.534360 11685 tablet_copy_client.cc:523] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:20.534401 7709 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:20.534549 7447 consensus_queue.cc:237] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } }
I20240501 08:00:20.534761 7159 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 21d76484f5dd499fb6fc3d534bffb779 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:00:20.535817 11685 tablet_bootstrap.cc:492] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap starting.
I20240501 08:00:20.536231 9318 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:20.537503 9184 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:20.537660 10383 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:20.538074 7168 catalog_manager.cc:5484] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 13 to 14, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:20.539731 11075 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:20.541628 7575 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:20.542683 10383 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:20.542846 9319 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:20.544365 7169 catalog_manager.cc:5484] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 6 to 7, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:20.545938 11154 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
I20240501 08:00:20.548110 9319 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
W20240501 08:00:20.548657 7382 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d18a2918c75467a9fe59979ce8936cd. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:20.549268 11781 tablet_bootstrap.cc:492] T 5970772c02d04e0b95a2f2feb7dcd655 P 3b69c5d799e14e438df760453438aa75: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:20.549228 7575 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } }
I20240501 08:00:20.550439 9449 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
I20240501 08:00:20.551338 7166 catalog_manager.cc:5484] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 12 to 13, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:20.554435 11781 tablet_bootstrap.cc:492] T 5970772c02d04e0b95a2f2feb7dcd655 P 3b69c5d799e14e438df760453438aa75: Bootstrap complete.
I20240501 08:00:20.554654 11781 ts_tablet_manager.cc:1367] T 5970772c02d04e0b95a2f2feb7dcd655 P 3b69c5d799e14e438df760453438aa75: Time spent bootstrapping tablet: real 0.077s user 0.027s sys 0.005s
I20240501 08:00:20.555816 11781 raft_consensus.cc:348] T 5970772c02d04e0b95a2f2feb7dcd655 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: true } }
I20240501 08:00:20.555950 11781 raft_consensus.cc:729] T 5970772c02d04e0b95a2f2feb7dcd655 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b69c5d799e14e438df760453438aa75, State: Initialized, Role: LEARNER
I20240501 08:00:20.556047 11781 consensus_queue.cc:260] T 5970772c02d04e0b95a2f2feb7dcd655 P 3b69c5d799e14e438df760453438aa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: true } }
I20240501 08:00:20.556317 11781 ts_tablet_manager.cc:1398] T 5970772c02d04e0b95a2f2feb7dcd655 P 3b69c5d799e14e438df760453438aa75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:20.557132 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session 3b69c5d799e14e438df760453438aa75-5970772c02d04e0b95a2f2feb7dcd655 received from {username='slave'} at 127.0.99.134:33617
I20240501 08:00:20.557215 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session 3b69c5d799e14e438df760453438aa75-5970772c02d04e0b95a2f2feb7dcd655 on tablet 5970772c02d04e0b95a2f2feb7dcd655 with peer 3b69c5d799e14e438df760453438aa75
I20240501 08:00:20.570972 7575 consensus_queue.cc:237] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:20.572954 9056 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:20.573323 7709 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:20.573908 11154 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:00:20.574390 7513 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8311d0921e9e417090e441884615525a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:20.574671 9319 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:20.575095 11851 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:20.576184 11823 ts_tablet_manager.cc:902] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:20.576637 11823 tablet_copy_client.cc:308] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:00:20.576671 11203 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:20.577073 7728 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 4dde9710a5eb4baa881d9dd04affe184 from peer 1c8cb9b1059a47cc822ca13a2b931776 ({username='slave'} at 127.0.99.135:43955)
I20240501 08:00:20.577188 7728 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 4dde9710a5eb4baa881d9dd04affe184 from peer 1c8cb9b1059a47cc822ca13a2b931776 at {username='slave'} at 127.0.99.135:43955: session id = 1c8cb9b1059a47cc822ca13a2b931776-4dde9710a5eb4baa881d9dd04affe184
I20240501 08:00:20.578305 7728 tablet_copy_source_session.cc:205] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:20.578557 11154 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.579197 11823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dde9710a5eb4baa881d9dd04affe184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:20.582228 11823 tablet_copy_client.cc:783] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:20.582401 11823 tablet_copy_client.cc:647] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:20.587517 7167 catalog_manager.cc:5484] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 7 to 8, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:20.590826 7709 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.592993 11823 tablet_copy_client.cc:523] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:20.595193 11823 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap starting.
I20240501 08:00:20.596406 9056 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.600210 9056 raft_consensus.cc:1203] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:00:20.603778 11685 tablet_bootstrap.cc:492] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:20.604203 11685 tablet_bootstrap.cc:492] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e: Bootstrap complete.
I20240501 08:00:20.604435 11685 ts_tablet_manager.cc:1367] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent bootstrapping tablet: real 0.069s user 0.023s sys 0.004s
I20240501 08:00:20.605077 11685 raft_consensus.cc:348] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:20.605288 11685 raft_consensus.cc:729] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1807c0c2e7d04b20b0cda311c42eef8e, State: Initialized, Role: LEARNER
I20240501 08:00:20.605448 11685 consensus_queue.cc:260] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:20.606289 11685 ts_tablet_manager.cc:1398] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:20.606743 7467 tablet_copy_service.cc:342] P d6b4be7bbd1649d0b7c5e8efe81f759f: Request end of tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-f647cfc562794e65a05918b241014d72 received from {username='slave'} at 127.0.99.136:53469
I20240501 08:00:20.606817 7467 tablet_copy_service.cc:433] P d6b4be7bbd1649d0b7c5e8efe81f759f: ending tablet copy session 1807c0c2e7d04b20b0cda311c42eef8e-f647cfc562794e65a05918b241014d72 on tablet f647cfc562794e65a05918b241014d72 with peer 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:20.610914 11203 raft_consensus.cc:1050] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 to VOTER
I20240501 08:00:20.611335 11203 consensus_queue.cc:237] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:20.613301 7709 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:20.613332 9056 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:20.614076 9319 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 08:00:20.614423 7513 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8311d0921e9e417090e441884615525a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:20.614751 10907 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:20.615370 11154 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:20.615625 11154 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:20.617136 9319 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.621928 10907 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.624879 7168 catalog_manager.cc:5484] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 8 to 9, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:20.625283 9056 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.625797 7709 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.632606 9319 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.636914 7575 consensus_queue.cc:237] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:20.638317 9056 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:20.638334 9319 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:20.638530 7709 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:20.638695 11154 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:20.638823 11851 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:20.639122 11851 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:20.640672 11203 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:20.641083 9056 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:20.642640 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8311d0921e9e417090e441884615525a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:20.643261 7167 catalog_manager.cc:5484] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 9 to 10, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:20.643333 7709 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:20.644529 9319 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:20.646473 9430 tablet_service.cc:1514] Processing DeleteTablet for tablet 8311d0921e9e417090e441884615525a with delete_type TABLET_DATA_TOMBSTONED (TS 1807c0c2e7d04b20b0cda311c42eef8e not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:58736
W20240501 08:00:20.647569 7160 catalog_manager.cc:4797] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): delete failed for tablet 8311d0921e9e417090e441884615525a because the tablet was not found. No further retry: Not found: Tablet not found: 8311d0921e9e417090e441884615525a
I20240501 08:00:20.649427 7575 consensus_queue.cc:237] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:20.650782 9056 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:20.651235 9319 raft_consensus.cc:1261] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:20.651541 11154 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:20.651859 11154 consensus_queue.cc:1035] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:20.653753 11880 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:20.654357 11823 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:20.654299 9056 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:20.654639 11823 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap complete.
I20240501 08:00:20.654801 11823 ts_tablet_manager.cc:1367] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent bootstrapping tablet: real 0.060s user 0.029s sys 0.000s
I20240501 08:00:20.655336 11823 raft_consensus.cc:348] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:20.655457 11823 raft_consensus.cc:729] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c8cb9b1059a47cc822ca13a2b931776, State: Initialized, Role: LEARNER
I20240501 08:00:20.655557 11823 consensus_queue.cc:260] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:20.655817 11823 ts_tablet_manager.cc:1398] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:20.656248 9318 raft_consensus.cc:2932] T 8311d0921e9e417090e441884615525a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:20.656503 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-4dde9710a5eb4baa881d9dd04affe184 received from {username='slave'} at 127.0.99.135:43955
I20240501 08:00:20.656584 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-4dde9710a5eb4baa881d9dd04affe184 on tablet 4dde9710a5eb4baa881d9dd04affe184 with peer 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:20.656988 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8311d0921e9e417090e441884615525a with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:20.657783 7168 catalog_manager.cc:5484] T 8311d0921e9e417090e441884615525a P d8aa5ac376a54e1f8417e5af11d28fa7 reported cstate change: config changed from index 10 to 11, VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:20.660704 7688 tablet_service.cc:1514] Processing DeleteTablet for tablet 8311d0921e9e417090e441884615525a with delete_type TABLET_DATA_TOMBSTONED (TS 09a7b6e2731a474b80411efb62323604 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:49464
I20240501 08:00:20.661152 11895 tablet_replica.cc:330] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604: stopping tablet replica
I20240501 08:00:20.661319 11895 raft_consensus.cc:2229] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:20.662431 11895 raft_consensus.cc:2258] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:20.663228 11895 ts_tablet_manager.cc:1875] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:20.665328 11895 ts_tablet_manager.cc:1888] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 08:00:20.665416 11895 log.cc:1196] T 8311d0921e9e417090e441884615525a P 09a7b6e2731a474b80411efb62323604: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/8311d0921e9e417090e441884615525a
I20240501 08:00:20.665895 7162 catalog_manager.cc:4830] TS 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789): tablet 8311d0921e9e417090e441884615525a (table rebalance_test_table_6 [id=22d40ca8e4714c2495974e28872c5687]) successfully deleted
I20240501 08:00:20.671063 7709 consensus_queue.cc:237] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:20.673065 7575 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:20.673441 9184 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:20.673491 11072 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:20.673962 11072 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:00:20.674142 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a37e631ae4f46a2be1ad5c43db5724b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:20.676671 11072 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:20.680173 9184 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:20.680546 7166 catalog_manager.cc:5484] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 6 to 7, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:20.681306 7575 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:20.695269 11769 tablet_bootstrap.cc:492] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:20.695781 11769 tablet_bootstrap.cc:492] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap complete.
I20240501 08:00:20.696130 11769 ts_tablet_manager.cc:1367] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent bootstrapping tablet: real 0.209s user 0.028s sys 0.003s
I20240501 08:00:20.699110 7447 consensus_queue.cc:237] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:20.699491 11769 raft_consensus.cc:348] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:20.700336 11769 raft_consensus.cc:729] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c8cb9b1059a47cc822ca13a2b931776, State: Initialized, Role: LEARNER
I20240501 08:00:20.700511 11769 consensus_queue.cc:260] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:20.701020 11769 ts_tablet_manager.cc:1398] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 08:00:20.701118 9056 raft_consensus.cc:1261] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:20.701846 7575 raft_consensus.cc:1261] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:20.702637 7316 raft_consensus.cc:1261] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:20.703110 11049 consensus_queue.cc:1035] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:20.703400 11075 consensus_queue.cc:1035] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:20.704370 11075 consensus_queue.cc:1035] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:20.704757 9449 raft_consensus.cc:1203] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:00:20.707202 11075 raft_consensus.cc:1013] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: attempt to promote peer d8aa5ac376a54e1f8417e5af11d28fa7: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 08:00:20.707556 7382 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8882528b26974c7aa73bb0ff0c444836. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:20.708703 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-7377810e08104526b8b8acdf886a3368 received from {username='slave'} at 127.0.99.135:36909
I20240501 08:00:20.708789 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-7377810e08104526b8b8acdf886a3368 on tablet 7377810e08104526b8b8acdf886a3368 with peer 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:20.708603 11045 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.711335 7575 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.711509 9056 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.715456 7172 catalog_manager.cc:5484] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.715968 7316 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.717536 10381 raft_consensus.cc:1050] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f: attempting to promote NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 to VOTER
I20240501 08:00:20.718614 10381 consensus_queue.cc:237] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:20.720499 7575 raft_consensus.cc:1261] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:20.721398 10381 consensus_queue.cc:1035] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:00:20.727908 7382 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8882528b26974c7aa73bb0ff0c444836. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:20.728416 7316 raft_consensus.cc:1261] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:20.728927 9056 raft_consensus.cc:1261] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:20.729122 10383 consensus_queue.cc:1035] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:20.730386 10383 consensus_queue.cc:1035] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:20.731465 9184 raft_consensus.cc:1203] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:00:20.735200 10381 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.735971 7575 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.737360 9056 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.738780 7316 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.739410 7167 catalog_manager.cc:5484] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 5 to 6, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:20.746178 7447 consensus_queue.cc:237] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:20.747839 9056 raft_consensus.cc:1261] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:20.747850 7316 raft_consensus.cc:1261] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:20.748374 10381 consensus_queue.cc:1035] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:20.748831 10381 consensus_queue.cc:1035] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:20.748893 7575 raft_consensus.cc:1261] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:20.749276 11045 consensus_queue.cc:1035] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:20.752336 10383 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.752509 7316 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.755064 7576 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.756551 7172 catalog_manager.cc:5484] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 6 to 7, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:20.758024 9056 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.758618 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8882528b26974c7aa73bb0ff0c444836 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:20.761564 9430 tablet_service.cc:1514] Processing DeleteTablet for tablet 8882528b26974c7aa73bb0ff0c444836 with delete_type TABLET_DATA_TOMBSTONED (TS 1807c0c2e7d04b20b0cda311c42eef8e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58736
I20240501 08:00:20.762301 7447 consensus_queue.cc:237] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
W20240501 08:00:20.763247 7160 catalog_manager.cc:4797] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): delete failed for tablet 8882528b26974c7aa73bb0ff0c444836 because the tablet was not found. No further retry: Not found: Tablet not found: 8882528b26974c7aa73bb0ff0c444836
I20240501 08:00:20.763859 9056 raft_consensus.cc:1261] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:20.764039 7576 raft_consensus.cc:1261] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:20.764309 11045 consensus_queue.cc:1035] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:20.765947 10381 consensus_queue.cc:1035] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:20.766636 11049 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.767671 9056 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.768735 9057 raft_consensus.cc:3037] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Advancing to term 2
I20240501 08:00:20.769733 9057 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 2 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:00:20.770385 7172 catalog_manager.cc:5484] T 8882528b26974c7aa73bb0ff0c444836 P d8aa5ac376a54e1f8417e5af11d28fa7 reported cstate change: config changed from index 7 to 8, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.770550 11072 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:20.772462 7709 raft_consensus.cc:1203] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 08:00:20.772897 9449 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 2 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:00:20.773149 7297 tablet_service.cc:1514] Processing DeleteTablet for tablet 8882528b26974c7aa73bb0ff0c444836 with delete_type TABLET_DATA_TOMBSTONED (TS b4f8020266b5455b872a2973fc396578 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35594
I20240501 08:00:20.773941 10335 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:20.776953 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8882528b26974c7aa73bb0ff0c444836 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:20.777848 11967 tablet_replica.cc:330] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578: stopping tablet replica
I20240501 08:00:20.778070 11967 raft_consensus.cc:2229] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:20.778385 7575 raft_consensus.cc:2932] T 8882528b26974c7aa73bb0ff0c444836 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.779042 11967 raft_consensus.cc:2258] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:20.779770 7576 consensus_queue.cc:237] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:20.781394 7709 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:20.781755 10907 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 08:00:20.782625 7512 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28baea6d7d3f442ba1dbfc5b37de90ac. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:20.783583 11880 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:20.784622 9184 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:20.785518 11851 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:20.786543 7709 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:20.789409 7172 catalog_manager.cc:5484] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:20.791339 7447 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 2 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:00:20.791903 10335 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:20.789811 9184 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:20.795800 11108 raft_consensus.cc:1050] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 to VOTER
I20240501 08:00:20.796159 11108 consensus_queue.cc:237] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:20.798694 9449 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 2 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:00:20.799218 9057 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 2 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:00:20.799750 11088 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:20.800261 11088 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:20.802064 7447 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 2 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:00:20.803396 10335 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:20.805563 10335 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.808288 9057 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.808678 7171 catalog_manager.cc:5484] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 7 to 9, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:20.808830 7447 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.782191 7313 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:20.816084 9449 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.818605 11154 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:20.819738 7709 consensus_queue.cc:237] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:20.820699 11660 ts_tablet_manager.cc:902] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:20.821154 11660 tablet_copy_client.cc:308] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:00:20.821393 9057 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 2 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:00:20.821751 7728 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet df6dd3d8482d4931b1d3014a67e739c6 from peer d8aa5ac376a54e1f8417e5af11d28fa7 ({username='slave'} at 127.0.99.133:35511)
I20240501 08:00:20.821849 7728 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet df6dd3d8482d4931b1d3014a67e739c6 from peer d8aa5ac376a54e1f8417e5af11d28fa7 at {username='slave'} at 127.0.99.133:35511: session id = d8aa5ac376a54e1f8417e5af11d28fa7-df6dd3d8482d4931b1d3014a67e739c6
I20240501 08:00:20.822185 7313 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:20.822825 7728 tablet_copy_source_session.cc:205] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:20.823693 11660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6dd3d8482d4931b1d3014a67e739c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:20.823699 11072 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:20.825722 11072 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.825964 11660 tablet_copy_client.cc:783] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:20.826146 11660 tablet_copy_client.cc:647] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:20.826063 9057 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.828522 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ffc23756b44d4d588e0a8175075d9bc6 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:20.829308 11154 raft_consensus.cc:1050] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER 3b69c5d799e14e438df760453438aa75 to VOTER
I20240501 08:00:20.829443 7170 catalog_manager.cc:5484] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:20.829823 11154 consensus_queue.cc:237] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:00:20.832432 7512 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28baea6d7d3f442ba1dbfc5b37de90ac. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:20.832589 9449 raft_consensus.cc:1261] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 2 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:00:20.833096 9184 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:20.833328 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc23756b44d4d588e0a8175075d9bc6 with delete_type TABLET_DATA_TOMBSTONED (TS d6b4be7bbd1649d0b7c5e8efe81f759f not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45222
I20240501 08:00:20.833683 11880 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:20.834256 7313 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:20.834337 11088 consensus_queue.cc:1035] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:20.834242 7709 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:20.835589 10558 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:20.835898 11880 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:20.836489 11660 tablet_copy_client.cc:523] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:20.836530 9449 raft_consensus.cc:2932] T ffc23756b44d4d588e0a8175075d9bc6 P 1807c0c2e7d04b20b0cda311c42eef8e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:20.837810 11880 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:20.838145 11660 tablet_bootstrap.cc:492] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap starting.
I20240501 08:00:20.841234 9184 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:20.841457 7174 catalog_manager.cc:5484] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 5 to 6, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:20.844246 7313 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:20.844444 7709 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:20.848291 7576 consensus_queue.cc:237] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:20.849581 11967 ts_tablet_manager.cc:1875] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:20.849725 9184 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 08:00:20.850033 7512 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28baea6d7d3f442ba1dbfc5b37de90ac. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:20.850306 11154 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:20.850440 7709 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:20.850853 11154 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:20.852643 10907 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:20.852888 11967 ts_tablet_manager.cc:1888] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:00:20.852963 11967 log.cc:1196] T 8882528b26974c7aa73bb0ff0c444836 P b4f8020266b5455b872a2973fc396578: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/8882528b26974c7aa73bb0ff0c444836
I20240501 08:00:20.853497 7162 catalog_manager.cc:4830] TS b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): tablet 8882528b26974c7aa73bb0ff0c444836 (table rebalance_test_table_7 [id=c7ddae9ad2184cb9bd097d68171c3fc4]) successfully deleted
I20240501 08:00:20.855036 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 28baea6d7d3f442ba1dbfc5b37de90ac with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:20.855489 9184 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:20.856315 7174 catalog_manager.cc:5484] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 6 to 7, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:20.857014 7707 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:00:20.859127 7297 tablet_service.cc:1514] Processing DeleteTablet for tablet 28baea6d7d3f442ba1dbfc5b37de90ac with delete_type TABLET_DATA_TOMBSTONED (TS b4f8020266b5455b872a2973fc396578 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35594
I20240501 08:00:20.859946 12001 tablet_replica.cc:330] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f: stopping tablet replica
I20240501 08:00:20.860560 12001 raft_consensus.cc:2229] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:20.864730 11880 raft_consensus.cc:1050] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER 3b69c5d799e14e438df760453438aa75 to VOTER
I20240501 08:00:20.865211 11880 consensus_queue.cc:237] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:20.867682 9184 raft_consensus.cc:1261] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:20.867720 7313 raft_consensus.cc:1261] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:20.868152 10558 consensus_queue.cc:1035] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:20.868547 11203 consensus_queue.cc:1035] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:20.874536 11967 tablet_replica.cc:330] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: stopping tablet replica
I20240501 08:00:20.874761 11967 raft_consensus.cc:2229] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:20.875082 11967 raft_consensus.cc:2258] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:20.866683 7448 raft_consensus.cc:1261] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:20.882395 12001 raft_consensus.cc:2258] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:20.882723 11203 consensus_queue.cc:1035] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:20.883198 12001 ts_tablet_manager.cc:1875] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:20.886265 11967 ts_tablet_manager.cc:1875] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:20.888264 11967 ts_tablet_manager.cc:1888] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:00:20.888339 11967 log.cc:1196] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/28baea6d7d3f442ba1dbfc5b37de90ac
I20240501 08:00:20.888689 7688 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:20.889001 7162 catalog_manager.cc:4830] TS b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): tablet 28baea6d7d3f442ba1dbfc5b37de90ac (table rebalance_test_table_9 [id=15bea2064e1c414fb23182f6bc324bfd]) successfully deleted
I20240501 08:00:20.891181 7688 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:20.891149 9188 raft_consensus.cc:1203] T 5970772c02d04e0b95a2f2feb7dcd655 P 3b69c5d799e14e438df760453438aa75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 08:00:20.896440 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:20.904129 10558 raft_consensus.cc:2932] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.907411 9184 raft_consensus.cc:2932] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.908640 7428 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:20.909535 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:20.910277 7172 catalog_manager.cc:5484] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:20.913636 7313 raft_consensus.cc:2932] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.914230 7448 raft_consensus.cc:2932] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3b69c5d799e14e438df760453438aa75 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.922356 7576 consensus_queue.cc:237] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:00:20.924023 7448 raft_consensus.cc:1261] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:20.925714 10907 consensus_queue.cc:1035] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:20.928033 9184 raft_consensus.cc:1261] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:20.928472 10907 consensus_queue.cc:1035] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:20.930577 10907 raft_consensus.cc:2932] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.932515 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4297f95243ac431099a8279c3d9e4b25 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:20.932624 9188 raft_consensus.cc:2932] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.935758 7171 catalog_manager.cc:5484] T 4297f95243ac431099a8279c3d9e4b25 P 3b69c5d799e14e438df760453438aa75 reported cstate change: config changed from index 5 to 6, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.938889 7297 tablet_service.cc:1514] Processing DeleteTablet for tablet 4297f95243ac431099a8279c3d9e4b25 with delete_type TABLET_DATA_TOMBSTONED (TS b4f8020266b5455b872a2973fc396578 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35594
I20240501 08:00:20.944083 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:20.945067 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:20.946331 12001 ts_tablet_manager.cc:1888] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 08:00:20.946503 12001 log.cc:1196] T ffc23756b44d4d588e0a8175075d9bc6 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ffc23756b44d4d588e0a8175075d9bc6
I20240501 08:00:20.950475 7159 catalog_manager.cc:4830] TS d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707): tablet ffc23756b44d4d588e0a8175075d9bc6 (table rebalance_test_table_8 [id=82bf711832394b1abda2824bb95c9358]) successfully deleted
I20240501 08:00:20.951826 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:20.951742 7448 raft_consensus.cc:2932] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } }
I20240501 08:00:20.954982 11967 tablet_replica.cc:330] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578: stopping tablet replica
I20240501 08:00:20.955369 11967 raft_consensus.cc:2229] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:20.955613 11967 raft_consensus.cc:2258] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:20.959427 11967 ts_tablet_manager.cc:1875] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:20.969916 11967 ts_tablet_manager.cc:1888] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:20.973706 7688 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:20.974166 11769 ts_tablet_manager.cc:902] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:20.974623 11967 log.cc:1196] T 4297f95243ac431099a8279c3d9e4b25 P b4f8020266b5455b872a2973fc396578: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4297f95243ac431099a8279c3d9e4b25
I20240501 08:00:20.974704 11769 tablet_copy_client.cc:308] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:20.975433 7162 catalog_manager.cc:4830] TS b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): tablet 4297f95243ac431099a8279c3d9e4b25 (table rebalance_test_table_9 [id=15bea2064e1c414fb23182f6bc324bfd]) successfully deleted
I20240501 08:00:20.975937 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet 21d76484f5dd499fb6fc3d534bffb779 from peer 1c8cb9b1059a47cc822ca13a2b931776 ({username='slave'} at 127.0.99.135:36909)
I20240501 08:00:20.976033 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet 21d76484f5dd499fb6fc3d534bffb779 from peer 1c8cb9b1059a47cc822ca13a2b931776 at {username='slave'} at 127.0.99.135:36909: session id = 1c8cb9b1059a47cc822ca13a2b931776-21d76484f5dd499fb6fc3d534bffb779
I20240501 08:00:20.976963 7599 tablet_copy_source_session.cc:205] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:20.977811 11769 ts_tablet_manager.cc:1875] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:00:20.979943 11769 ts_tablet_manager.cc:1888] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.12
I20240501 08:00:20.980105 11769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d76484f5dd499fb6fc3d534bffb779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:20.981451 11769 tablet_copy_client.cc:783] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:20.981644 11769 tablet_copy_client.cc:647] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:20.987526 9449 raft_consensus.cc:1203] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:00:21.004566 9319 raft_consensus.cc:1203] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:00:21.006772 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:21.014294 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:21.020007 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:21.022392 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:21.026226 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:21.033944 7428 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:21.035409 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:21.041249 11769 tablet_copy_client.cc:523] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:21.043510 11769 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap starting.
I20240501 08:00:21.046005 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:21.047159 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:21.058900 7313 consensus_queue.cc:237] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:21.060518 7448 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:00:21.061024 11143 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:00:21.062601 11660 tablet_bootstrap.cc:492] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:21.063370 9319 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:00:21.063839 11660 tablet_bootstrap.cc:492] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap complete.
I20240501 08:00:21.063803 11143 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:00:21.065218 11660 ts_tablet_manager.cc:1367] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent bootstrapping tablet: real 0.227s user 0.030s sys 0.001s
W20240501 08:00:21.065703 7251 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1088a116ccec4020aec562a69791a188. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:21.065757 11660 raft_consensus.cc:348] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:21.066118 11660 raft_consensus.cc:729] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8aa5ac376a54e1f8417e5af11d28fa7, State: Initialized, Role: LEARNER
I20240501 08:00:21.066273 11660 consensus_queue.cc:260] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:21.067860 11660 ts_tablet_manager.cc:1398] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:00:21.068636 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-df6dd3d8482d4931b1d3014a67e739c6 received from {username='slave'} at 127.0.99.133:35511
I20240501 08:00:21.068724 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-df6dd3d8482d4931b1d3014a67e739c6 on tablet df6dd3d8482d4931b1d3014a67e739c6 with peer d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:21.073820 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:21.086294 11143 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:21.089999 7170 catalog_manager.cc:5484] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 11 to 12, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:21.091728 9319 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:21.098541 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:21.099398 11769 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:21.099709 11769 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap complete.
I20240501 08:00:21.099874 11769 ts_tablet_manager.cc:1367] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent bootstrapping tablet: real 0.056s user 0.026s sys 0.001s
I20240501 08:00:21.100335 11769 raft_consensus.cc:348] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:21.100466 11769 raft_consensus.cc:729] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c8cb9b1059a47cc822ca13a2b931776, State: Initialized, Role: LEARNER
I20240501 08:00:21.100555 11769 consensus_queue.cc:260] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:21.100819 11769 ts_tablet_manager.cc:1398] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:21.101423 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-21d76484f5dd499fb6fc3d534bffb779 received from {username='slave'} at 127.0.99.135:36909
I20240501 08:00:21.101526 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-21d76484f5dd499fb6fc3d534bffb779 on tablet 21d76484f5dd499fb6fc3d534bffb779 with peer 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:21.103799 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:21.115448 9319 raft_consensus.cc:1203] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 08:00:21.122062 7428 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:21.125360 11660 ts_tablet_manager.cc:902] T 5a37e631ae4f46a2be1ad5c43db5724b P d8aa5ac376a54e1f8417e5af11d28fa7: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:21.126106 11660 tablet_copy_client.cc:308] T 5a37e631ae4f46a2be1ad5c43db5724b P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:00:21.126597 7728 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 5a37e631ae4f46a2be1ad5c43db5724b from peer d8aa5ac376a54e1f8417e5af11d28fa7 ({username='slave'} at 127.0.99.133:35511)
I20240501 08:00:21.126695 7728 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 5a37e631ae4f46a2be1ad5c43db5724b from peer d8aa5ac376a54e1f8417e5af11d28fa7 at {username='slave'} at 127.0.99.133:35511: session id = d8aa5ac376a54e1f8417e5af11d28fa7-5a37e631ae4f46a2be1ad5c43db5724b
I20240501 08:00:21.127575 7728 tablet_copy_source_session.cc:205] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:21.129216 11660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a37e631ae4f46a2be1ad5c43db5724b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:21.129935 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:21.141096 7448 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:21.146509 9168 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:21.160228 11660 tablet_copy_client.cc:783] T 5a37e631ae4f46a2be1ad5c43db5724b P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:21.160446 11660 tablet_copy_client.cc:647] T 5a37e631ae4f46a2be1ad5c43db5724b P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:21.168942 12073 ts_tablet_manager.cc:902] T 5d18a2918c75467a9fe59979ce8936cd P 3b69c5d799e14e438df760453438aa75: Initiating tablet copy from peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
F20240501 08:00:21.169281 12073 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-02b42b5fd3d440ea950b35ed461b8e12.queue_time_us, tablet-download-pool-02b42b5fd3d440ea950b35ed461b8e12.queue_time_us) (tablet-download-pool-02b42b5fd3d440ea950b35ed461b8e12.run_cpu_time_us, tablet-download-pool-02b42b5fd3d440ea950b35ed461b8e12.run_cpu_time_us) (tablet-download-pool-02b42b5fd3d440ea950b35ed461b8e12.run_wall_time_us, tablet-download-pool-02b42b5fd3d440ea950b35ed461b8e12.run_wall_time_us) (tablet-download-pool-07967f68a412411494c6dd1f47209009.queue_time_us, tablet-download-pool-07967f68a412411494c6dd1f47209009.queue_time_us) (tablet-download-pool-07967f68a412411494c6dd1f47209009.run_cpu_time_us, tablet-download-pool-07967f68a412411494c6dd1f47209009.run_cpu_time_us) (tablet-download-pool-07967f68a412411494c6dd1f47209009.run_wall_time_us, tablet-download-pool-07967f68a412411494c6dd1f47209009.run_wall_time_us) (tablet-download-pool-1088a116ccec4020aec562a69791a188.queue_time_us, tablet-download-pool-1088a116ccec4020aec562a69791a188.queue_time_us) (tablet-download-pool-1088a116ccec4020aec562a69791a188.run_cpu_time_us, tablet-download-pool-1088a116ccec4020aec562a69791a188.run_cpu_time_us) (tablet-download-pool-1088a116ccec4020aec562a69791a188.run_wall_time_us, tablet-download-pool-1088a116ccec4020aec562a69791a188.run_wall_time_us) (tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.queue_time_us, tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.queue_time_us) (tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.run_cpu_time_us, tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.run_cpu_time_us) (tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.run_wall_time_us, tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.run_wall_time_us) (tablet-download-pool-28baea6d7d3f442ba1dbfc5b37de90ac.queue_time_us, tablet-download-pool-28baea6d7d3f442ba1dbfc5b37de90ac.queue_time_us) (tablet-download-pool-28baea6d7d3f442ba1dbfc5b37de90ac.run_cpu_time_us, tablet-download-pool-28baea6d7d3f442ba1dbfc5b37de90ac.run_cpu_time_us) (tablet-download-pool-28baea6d7d3f442ba1dbfc5b37de90ac.run_wall_time_us, tablet-download-pool-28baea6d7d3f442ba1dbfc5b37de90ac.run_wall_time_us) (tablet-download-pool-3d8ae3c82c9d4c5c858fa3f9d5959eb7.queue_time_us, tablet-download-pool-3d8ae3c82c9d4c5c858fa3f9d5959eb7.queue_time_us) (tablet-download-pool-3d8ae3c82c9d4c5c858fa3f9d5959eb7.run_cpu_time_us, tablet-download-pool-3d8ae3c82c9d4c5c858fa3f9d5959eb7.run_cpu_time_us) (tablet-download-pool-3d8ae3c82c9d4c5c858fa3f9d5959eb7.run_wall_time_us, tablet-download-pool-3d8ae3c82c9d4c5c858fa3f9d5959eb7.run_wall_time_us) (tablet-download-pool-4297f95243ac431099a8279c3d9e4b25.queue_time_us, tablet-download-pool-4297f95243ac431099a8279c3d9e4b25.queue_time_us) (tablet-download-pool-4297f95243ac431099a8279c3d9e4b25.run_cpu_time_us, tablet-download-pool-4297f95243ac431099a8279c3d9e4b25.run_cpu_time_us) (tablet-download-pool-4297f95243ac431099a8279c3d9e4b25.run_wall_time_us, tablet-download-pool-4297f95243ac431099a8279c3d9e4b25.run_wall_time_us) (tablet-download-pool-4ea0db42a75a4879b3d3d608666dc2bf.queue_time_us, tablet-download-pool-4ea0db42a75a4879b3d3d608666dc2bf.queue_time_us) (tablet-download-pool-4ea0db42a75a4879b3d3d608666dc2bf.run_cpu_time_us, tablet-download-pool-4ea0db42a75a4879b3d3d608666dc2bf.run_cpu_time_us) (tablet-download-pool-4ea0db42a75a4879b3d3d608666dc2bf.run_wall_time_us, tablet-download-pool-4ea0db42a75a4879b3d3d608666dc2bf.run_wall_time_us) (tablet-download-pool-5970772c02d04e0b95a2f2feb7dcd655.queue_time_us, tablet-download-pool-5970772c02d04e0b95a2f2feb7dcd655.queue_time_us) (tablet-download-pool-5970772c02d04e0b95a2f2feb7dcd655.run_cpu_time_us, tablet-download-pool-5970772c02d04e0b95a2f2feb7dcd655.run_cpu_time_us) (tablet-download-pool-5970772c02d04e0b95a2f2feb7dcd655.run_wall_time_us, tablet-download-pool-5970772c02d04e0b95a2f2feb7dcd655.run_wall_time_us) (tablet-download-pool-5a37e631ae4f46a2be1ad5c43db5724b.queue_time_us, tablet-download-pool-5a37e631ae4f46a2be1ad5c43db5724b.queue_time_us) (tablet-download-pool-5a37e631ae4f46a2be1ad5c43db5724b.run_cpu_time_us, tablet-download-pool-5a37e631ae4f46a2be1ad5c43db5724b.run_cpu_time_us) (tablet-download-pool-5a37e631ae4f46a2be1ad5c43db5724b.run_wall_time_us, tablet-download-pool-5a37e631ae4f46a2be1ad5c43db5724b.run_wall_time_us) (tablet-download-pool-5d18a2918c75467a9fe59979ce8936cd.queue_time_us, tablet-download-pool-5d18a2918c75467a9fe59979ce8936cd.queue_time_us) (tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.queue_time_us, tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.queue_time_us) (tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.run_cpu_time_us, tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.run_cpu_time_us) (tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.run_wall_time_us, tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.run_wall_time_us) (tablet-download-pool-b8fdc2b9c2ef44b0bfec59a1a2f6b626.queue_time_us, tablet-download-pool-b8fdc2b9c2ef44b0bfec59a1a2f6b626.queue_time_us) (tablet-download-pool-b8fdc2b9c2ef44b0bfec59a1a2f6b626.run_cpu_time_us, tablet-download-pool-b8fdc2b9c2ef44b0bfec59a1a2f6b626.run_cpu_time_us) (tablet-download-pool-b8fdc2b9c2ef44b0bfec59a1a2f6b626.run_wall_time_us, tablet-download-pool-b8fdc2b9c2ef44b0bfec59a1a2f6b626.run_wall_time_us) (tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.queue_time_us, tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.queue_time_us) (tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.run_cpu_time_us, tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.run_cpu_time_us) (tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.run_wall_time_us, tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.run_wall_time_us) (tablet-download-pool-f647cfc562794e65a05918b241014d72.queue_time_us, tablet-download-pool-f647cfc562794e65a05918b241014d72.queue_time_us) (tablet-download-pool-f647cfc562794e65a05918b241014d72.run_cpu_time_us, tablet-download-pool-f647cfc562794e65a05918b241014d72.run_cpu_time_us) (tablet-download-pool-f647cfc562794e65a05918b241014d72.run_wall_time_us, tablet-download-pool-f647cfc562794e65a05918b241014d72.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f453b79bdcd google::LogMessage::Fail() at ??:0
@ 0x7f453b79fb93 google::LogMessage::SendToLog() at ??:0
I20240501 08:00:21.178045 11660 tablet_copy_client.cc:523] T 5a37e631ae4f46a2be1ad5c43db5724b P d8aa5ac376a54e1f8417e5af11d28fa7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f453b79b7cc google::LogMessage::Flush() at ??:0
I20240501 08:00:21.179594 12072 ts_tablet_manager.cc:902] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:21.180351 12072 tablet_copy_client.cc:308] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:21.181043 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet bde42e485c4b49ce8acd694a25ba683a from peer b4f8020266b5455b872a2973fc396578 ({username='slave'} at 127.0.99.129:41265)
I20240501 08:00:21.181164 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet bde42e485c4b49ce8acd694a25ba683a from peer b4f8020266b5455b872a2973fc396578 at {username='slave'} at 127.0.99.129:41265: session id = b4f8020266b5455b872a2973fc396578-bde42e485c4b49ce8acd694a25ba683a
@ 0x7f453b79cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 08:00:21.182974 7599 tablet_copy_source_session.cc:205] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:21.184139 12072 ts_tablet_manager.cc:1875] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_COPYING
@ 0x7f453cc05044 kudu::TraceMetrics::InternName() at ??:0
I20240501 08:00:21.187628 12072 ts_tablet_manager.cc:1888] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.10
I20240501 08:00:21.187821 12072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde42e485c4b49ce8acd694a25ba683a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:21.189203 12072 tablet_copy_client.cc:783] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:21.189358 12072 tablet_copy_client.cc:647] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:21.190457 11660 tablet_bootstrap.cc:492] T 5a37e631ae4f46a2be1ad5c43db5724b P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap starting.
@ 0x7f453cbe9f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f453cbe7efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f453f79a77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f453f7a3a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 08:00:21.207883 12072 tablet_copy_client.cc:523] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:21.210296 11143 raft_consensus.cc:1050] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e to VOTER
@ 0x7f453f82476f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 08:00:21.210708 11143 consensus_queue.cc:237] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:21.212205 9057 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:21.212721 10080 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:21.214008 7576 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:21.214594 9449 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:21.216104 7313 consensus_queue.cc:237] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } }
I20240501 08:00:21.217365 12062 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
@ 0x7f453f8384f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 08:00:21.218611 9319 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:21.219501 9056 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:21.219871 10306 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:21.220180 10306 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
@ 0x7f453f823226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240501 08:00:21.221508 10080 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
@ 0x7f453f832c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x555d4fe7375c std::function<>::operator()() at ??:0
W20240501 08:00:21.226433 7250 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861): Couldn't send request to peer 31b02f06be8a4455b94b1254d77005bd. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:21.226831 10080 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } } }
@ 0x7f453cbece06 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 08:00:21.229952 7168 catalog_manager.cc:5484] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 9 to 10, NON_VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
@ 0x7f453cbed6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240501 08:00:21.231802 9056 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } } }
I20240501 08:00:21.233794 9319 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } } }
@ 0x7f453cbef03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x555d4fe7375c std::function<>::operator()() at ??:0
@ 0x7f453cbdc807 kudu::Thread::SuperviseThread() at ??:0
I20240501 08:00:21.239691 12072 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
@ 0x7f453d41a6db start_thread at ??:0
@ 0x7f453a95271f clone at ??:0
I20240501 08:00:21.253681 12104 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:21.270423 7688 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:21.279490 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:21.281708 9447 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:21.282611 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:21.284466 7170 catalog_manager.cc:5484] T b357c740aca144b7b063845726f419e2 P 1807c0c2e7d04b20b0cda311c42eef8e reported cstate change: config changed from index 7 to 8, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:21.285493 9056 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:21.288273 7576 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:21.308223 12072 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:21.308570 12072 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: Bootstrap complete.
I20240501 08:00:21.308741 12072 ts_tablet_manager.cc:1367] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.069s user 0.030s sys 0.004s
I20240501 08:00:21.309180 12072 raft_consensus.cc:348] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
I20240501 08:00:21.309309 12072 raft_consensus.cc:729] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: LEARNER
I20240501 08:00:21.309410 12072 consensus_queue.cc:260] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
I20240501 08:00:21.309685 12072 ts_tablet_manager.cc:1398] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 08:00:21.310313 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session b4f8020266b5455b872a2973fc396578-bde42e485c4b49ce8acd694a25ba683a received from {username='slave'} at 127.0.99.129:41265
I20240501 08:00:21.310405 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session b4f8020266b5455b872a2973fc396578-bde42e485c4b49ce8acd694a25ba683a on tablet bde42e485c4b49ce8acd694a25ba683a with peer b4f8020266b5455b872a2973fc396578
I20240501 08:00:21.316958 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:21.319054 7428 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:21.322947 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:21.329725 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:21.330269 9056 raft_consensus.cc:1203] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:00:21.353181 11660 tablet_bootstrap.cc:492] T 5a37e631ae4f46a2be1ad5c43db5724b P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:21.353493 11660 tablet_bootstrap.cc:492] T 5a37e631ae4f46a2be1ad5c43db5724b P d8aa5ac376a54e1f8417e5af11d28fa7: Bootstrap complete.
I20240501 08:00:21.353662 11660 ts_tablet_manager.cc:1367] T 5a37e631ae4f46a2be1ad5c43db5724b P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent bootstrapping tablet: real 0.163s user 0.022s sys 0.004s
I20240501 08:00:21.354138 11660 raft_consensus.cc:348] T 5a37e631ae4f46a2be1ad5c43db5724b P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:21.354274 11660 raft_consensus.cc:729] T 5a37e631ae4f46a2be1ad5c43db5724b P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8aa5ac376a54e1f8417e5af11d28fa7, State: Initialized, Role: LEARNER
I20240501 08:00:21.354367 11660 consensus_queue.cc:260] T 5a37e631ae4f46a2be1ad5c43db5724b P d8aa5ac376a54e1f8417e5af11d28fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:21.355178 11660 ts_tablet_manager.cc:1398] T 5a37e631ae4f46a2be1ad5c43db5724b P d8aa5ac376a54e1f8417e5af11d28fa7: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 08:00:21.355660 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-5a37e631ae4f46a2be1ad5c43db5724b received from {username='slave'} at 127.0.99.133:35511
I20240501 08:00:21.355742 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session d8aa5ac376a54e1f8417e5af11d28fa7-5a37e631ae4f46a2be1ad5c43db5724b on tablet 5a37e631ae4f46a2be1ad5c43db5724b with peer d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:21.375802 7313 consensus_queue.cc:237] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } }
I20240501 08:00:21.377578 9057 raft_consensus.cc:1261] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240501 08:00:21.378108 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02b42b5fd3d440ea950b35ed461b8e12. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:21.378492 10306 consensus_queue.cc:1035] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:00:21.387512 10306 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:21.387873 9057 raft_consensus.cc:2932] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:21.391837 7172 catalog_manager.cc:5484] T 02b42b5fd3d440ea950b35ed461b8e12 P d8aa5ac376a54e1f8417e5af11d28fa7 reported cstate change: config changed from index 11 to 12, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: true } } }
I20240501 08:00:21.394445 12062 raft_consensus.cc:1050] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER 09a7b6e2731a474b80411efb62323604 to VOTER
I20240501 08:00:21.394902 12062 consensus_queue.cc:237] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } }
I20240501 08:00:21.397058 7448 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:21.397058 9319 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:21.397514 11143 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:21.397630 10080 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:21.397917 7709 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:21.400686 11143 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:21.401782 10080 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 09a7b6e2731a474b80411efb62323604 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
I20240501 08:00:21.402101 7448 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 09a7b6e2731a474b80411efb62323604 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
I20240501 08:00:21.404053 9319 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 09a7b6e2731a474b80411efb62323604 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
I20240501 08:00:21.407322 7168 catalog_manager.cc:5484] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 8 to 9, 09a7b6e2731a474b80411efb62323604 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:21.408526 7576 consensus_queue.cc:237] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: true } }
I20240501 08:00:21.410137 9057 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:21.410811 11154 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:21.411234 9447 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 08:00:21.411286 7515 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789): Couldn't send request to peer 09a7b6e2731a474b80411efb62323604. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:21.411741 11880 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:21.420442 9319 raft_consensus.cc:1203] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.13->[1.14-1.14] Dedup: 1.14->[]
I20240501 08:00:21.421933 7709 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 09a7b6e2731a474b80411efb62323604 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
I20240501 08:00:21.426815 7313 consensus_queue.cc:237] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } }
I20240501 08:00:21.426927 11851 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: true } } }
I20240501 08:00:21.429211 9319 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:21.429275 7448 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:21.429771 10080 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:21.430578 7168 catalog_manager.cc:5484] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 9 to 10, NON_VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:21.430789 10306 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:21.431603 9057 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: true } } }
I20240501 08:00:21.434301 7709 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:00:21.434818 10306 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:21.434770 12062 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
I20240501 08:00:21.435763 9319 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
I20240501 08:00:21.438050 7174 catalog_manager.cc:5484] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 9 to 10, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:21.438858 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5970772c02d04e0b95a2f2feb7dcd655 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:21.438815 7709 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
I20240501 08:00:21.439339 7448 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
I20240501 08:00:21.442674 9447 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 09a7b6e2731a474b80411efb62323604 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: true } } }
I20240501 08:00:21.444912 11203 raft_consensus.cc:1050] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 to VOTER
I20240501 08:00:21.445329 11203 consensus_queue.cc:237] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:21.447351 9447 raft_consensus.cc:1261] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:21.447795 10907 consensus_queue.cc:1035] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:21.449196 7448 raft_consensus.cc:1261] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:21.449649 10907 consensus_queue.cc:1035] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:21.451872 11769 ts_tablet_manager.cc:902] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:21.452350 11769 tablet_copy_client.cc:308] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:00:21.455145 9447 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:21.452996 9319 raft_consensus.cc:1261] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:21.493600 7168 catalog_manager.cc:5484] T 7377810e08104526b8b8acdf886a3368 P 1807c0c2e7d04b20b0cda311c42eef8e reported cstate change: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:21.509609 9430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:21.511317 10335 raft_consensus.cc:1050] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 to VOTER
I20240501 08:00:21.511761 10335 consensus_queue.cc:237] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:21.513005 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:21.515761 7688 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:21.539603 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:21.514094 9319 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:21.514032 9447 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:21.560799 12259 ts_tablet_manager.cc:902] T 1088a116ccec4020aec562a69791a188 P 1807c0c2e7d04b20b0cda311c42eef8e: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
F20240501 08:00:21.561012 12259 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raI20240501 08:00:21.561517 12248 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:21.566535 12247 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:21.572239 12248 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:21.572585 9318 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
ft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1088a116ccec4020aec562a69791a188.queue_time_us, tablet-download-pool-1088a116ccec4020aec562a69791a188.queue_time_us) (tablet-download-pool-1e44704a9b9c46cf954d8e6937983841.queue_time_us, tablet-download-pool-1e44704a9b9c46cf954d8e6937983841.queue_time_us) (tablet-download-pool-1e44704a9b9c46cf954d8e6937983841.run_cpu_time_us, tablet-download-pool-1e44704a9b9c46cf954d8e6937983841.run_cpu_time_us) (tablet-download-pool-1e44704a9b9c46cf954d8e6937983841.run_wall_time_us, tablet-download-pool-1e44704a9b9c46cf954d8e6937983841.run_wall_time_us) (tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.queue_time_us, tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.queue_time_us) (tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.run_cpu_time_us, tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.run_cpu_time_us) (tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.run_wall_time_us, tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.run_wall_time_us) (tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.queue_time_us, tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.queue_time_us) (tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.run_cpu_time_us, tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.run_cpu_time_us) (tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.run_wall_time_us, tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.run_wall_time_us) (tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.queue_time_us, tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.queue_time_us) (tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.run_cpu_time_us, tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.run_cpu_time_us) (tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.run_wall_time_us, tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.run_wall_time_us) (tablet-download-pool-4ea0db42a75a4879b3d3d608666dc2bf.queue_time_us, tablet-download-pool-4ea0db42a75a4879b3d3d608666dc2bf.queue_time_us) (tablet-download-pool-4ea0db42a75a4879b3d3d608666dc2bf.run_cpu_time_us, tablet-download-pool-4ea0db42a75a4879b3d3d608666dc2bf.run_cpu_time_us) (tablet-download-pool-4ea0db42a75a4879b3d3d608666dc2bf.run_wall_time_us, tablet-download-pool-4ea0db42a75a4879b3d3d608666dc2bf.run_wall_time_us) (tablet-download-pool-7377810e08104526b8b8acdf886a3368.queue_time_us, tablet-download-pool-7377810e08104526b8b8acdf886a3368.queue_time_us) (tablet-download-pool-7377810e08104526b8b8acdf886a3368.run_cpu_time_us, tablet-download-pool-7377810e08104526b8b8acdf886a3368.run_cpu_time_us) (tablet-download-pool-7377810e08104526b8b8acdf886a3368.run_wall_time_us, tablet-download-pool-7377810e08104526b8b8acdf886a3368.run_wall_time_us) (tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.queue_time_us, tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.queue_time_us) (tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.run_cpu_time_us, tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.run_cpu_time_us) (tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.run_wall_time_us, tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.run_wall_time_us) (tablet-download-pool-b357c740aca144b7b063845726f419e2.queue_time_us, tablet-download-pool-b357c740aca144b7b063845726f419e2.queue_time_us) (tablet-download-pool-b357c740aca144b7b063845726f419e2.run_cpu_time_us, tablet-download-pool-b357c740aca144b7b063845726f419e2.run_cpu_time_us) (tablet-download-pool-b357c740aca144b7b063845726f419e2.run_wall_time_us, tablet-download-pool-b357c740aca144b7b063845726f419e2.run_wall_time_us) (tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.queue_time_us, tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.queue_time_us) (tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.run_cpu_time_us, tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.run_cpu_time_us) (tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.run_wall_time_us, tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.run_wall_time_us) (tablet-download-pool-cc6f7dcc930e4937865d37cb3c3bb3d2.queue_time_us, tablet-download-pool-cc6f7dcc930e4937865d37cb3c3bb3d2.queue_time_us) (tablet-download-pool-cc6f7dcc930e4937865d37cb3c3bb3d2.run_cpu_time_us, tablet-download-pool-cc6f7dcc930e4937865d37cb3c3bb3d2.run_cpu_time_us) (tablet-download-pool-cc6f7dcc930e4937865d37cb3c3bb3d2.run_wall_time_us, tablet-download-pool-cc6f7dcc930e4937865d37cb3c3bb3d2.run_wall_time_us) (tablet-download-pool-cd07a00b6817400f86bf8c32bd592d8b.queue_time_us, tablet-download-pool-cd07a00b6817400f86bf8c32bd592d8b.queue_time_us) (tablet-download-pool-cd07a00b6817400f86bf8c32bd592d8b.run_cpu_time_us, tablet-download-pool-cd07a00b6817400f86bf8c32bd592d8b.run_cpu_time_us) (tablet-download-pool-cd07a00b6817400f86bf8c32bd592d8b.run_wall_time_us, tablet-download-pool-cd07a00b6817400f86bf8c32bd592d8b.run_wall_time_us) (tablet-download-pool-dd17760cd5c2454db97610b2f71c0ee2.queue_time_us, tablet-download-pool-dd17760cd5c2454db97610b2f71c0ee2.queue_time_us) (tablet-download-pool-dd17760cd5c2454db97610b2f71c0ee2.run_cpu_time_us, tablet-download-pool-dd17760cd5c2454db97610b2f71c0ee2.run_cpu_time_us) (tablet-download-pool-dd17760cd5c2454db97610b2f71c0ee2.run_wall_time_us, tablet-download-pool-dd17760cd5c2454db97610b2f71c0ee2.run_wall_time_us) (tablet-download-pool-f647cfc562794e65a05918b241014d72.queue_time_us, tablet-download-pool-f647cfc562794e65a05918b241014d72.queue_time_us) (tablet-download-pool-f647cfc562794e65a05918b241014d72.run_cpu_time_us, tablet-download-pool-f647cfc562794e65a05918b241014d72.run_cpu_time_us) (tablet-download-pool-f647cfc562794e65a05918b241014d72.run_wall_time_us, tablet-download-pool-f647cfc562794e65a05918b241014d72.run_wall_time_us) (tablet-download-pool-ffc23756b44d4d588e0a8175075d9bc6.queue_time_us, tablet-download-pool-ffc23756b44d4d588e0a8175075d9bc6.queue_time_us) (tablet-download-pool-ffc23756b44d4d588e0a8175075d9bc6.run_cpu_time_us, tablet-download-pool-ffc23756b44d4d588e0a8175075d9bc6.run_cpu_time_us) (tablet-download-pool-ffc23756b44d4d588e0a8175075d9bc6.run_wall_time_us, tablet-download-pool-ffc23756b44d4d588e0a8175075d9bc6.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 08:00:21.624214 7313 raft_consensus.cc:1203] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
W20240501 08:00:21.627696 7251 connection.cc:535] client connection to 127.0.99.134:33589 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:21.640270 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0x7ff1b3ba5dcd google::LogMessage::Fail() at ??:0
W20240501 08:00:21.641826 7251 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:21.575634 7168 catalog_manager.cc:5484] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 10 to 11, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:00:21.629402 7160 connection.cc:535] client connection to 127.0.99.134:33589 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:21.642184 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:21.642413 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:21.642467 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:21.451598 7447 consensus_queue.cc:237] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } }
I20240501 08:00:21.526934 11075 raft_consensus.cc:1050] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f: attempting to promote NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e to VOTER
I20240501 08:00:21.513718 9057 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:21.643314 11075 consensus_queue.cc:237] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:00:21.453922 11154 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:21.577136 9447 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
W20240501 08:00:21.627465 7382 connection.cc:535] client connection to 127.0.99.134:33589 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:21.645201 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:00:21.646230 9449 raft_consensus.cc:1261] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:21.646262 9450 raft_consensus.cc:1261] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:21.646656 11075 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:21.646998 11075 consensus_queue.cc:1035] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:21.519620 9056 raft_consensus.cc:1203] T 5a37e631ae4f46a2be1ad5c43db5724b P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:00:21.492890 11851 consensus_queue.cc:1035] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:21.492973 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet 28baea6d7d3f442ba1dbfc5b37de90ac from peer 1c8cb9b1059a47cc822ca13a2b931776 ({username='slave'} at 127.0.99.135:36909)
W20240501 08:00:21.627715 7513 connection.cc:535] client connection to 127.0.99.134:33589 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:00:21.649188 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet 28baea6d7d3f442ba1dbfc5b37de90ac from peer 1c8cb9b1059a47cc822ca13a2b931776 at {username='slave'} at 127.0.99.135:36909: session id = 1c8cb9b1059a47cc822ca13a2b931776-28baea6d7d3f442ba1dbfc5b37de90ac
W20240501 08:00:21.649256 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:00:21.649497 9318 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:21.650082 7599 tablet_copy_source_session.cc:205] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:21.629454 12302 ts_tablet_manager.cc:902] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:21.650936 12302 tablet_copy_client.cc:308] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:21.651929 7575 raft_consensus.cc:1261] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 08:00:21.652395 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861): Couldn't send request to peer 31b02f06be8a4455b94b1254d77005bd. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:21.652572 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet e178f7c3c17c4e1ca3d16b6441f76e77 from peer 31b02f06be8a4455b94b1254d77005bd ({username='slave'} at 127.0.99.131:52783)
I20240501 08:00:21.652683 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet e178f7c3c17c4e1ca3d16b6441f76e77 from peer 31b02f06be8a4455b94b1254d77005bd at {username='slave'} at 127.0.99.131:52783: session id = 31b02f06be8a4455b94b1254d77005bd-e178f7c3c17c4e1ca3d16b6441f76e77
I20240501 08:00:21.653369 7428 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:21.653622 7336 tablet_copy_source_session.cc:205] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:21.653831 11769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28baea6d7d3f442ba1dbfc5b37de90ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:21.654723 11075 consensus_queue.cc:1035] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:21.655117 12302 ts_tablet_manager.cc:1875] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:00:21.655164 11049 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } } }
I20240501 08:00:21.656041 11769 tablet_copy_client.cc:783] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:21.655908 9450 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } } }
I20240501 08:00:21.656189 11769 tablet_copy_client.cc:647] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:21.657403 12302 ts_tablet_manager.cc:1888] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240501 08:00:21.657581 12302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e178f7c3c17c4e1ca3d16b6441f76e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:21.658329 7174 catalog_manager.cc:5484] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 6 to 7, NON_VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:21.659901 12302 tablet_copy_client.cc:783] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:21.660063 12302 tablet_copy_client.cc:647] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: tablet copy: Starting download of 1 WAL segments...
W20240501 08:00:21.660790 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:00:21.661437 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:21.662554 12302 tablet_copy_client.cc:523] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:21.662685 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:21.664436 12302 tablet_bootstrap.cc:492] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:21.667111 7446 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:21.617884 7709 consensus_queue.cc:237] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
W20240501 08:00:21.627575 7644 connection.cc:535] client connection to 127.0.99.134:33589 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:21.672657 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:00:21.646847 7707 raft_consensus.cc:1261] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:21.674561 9318 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:00:21.675956 11088 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:00:21.676070 12313 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:21.676950 9450 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:00:21.677276 7707 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } } }
I20240501 08:00:21.677592 11088 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:00:21.677558 11075 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
W20240501 08:00:21.679623 7646 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): Couldn't send request to peer b4f8020266b5455b872a2973fc396578. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:21.679699 9449 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:21.679824 7575 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:21.682384 11088 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
I20240501 08:00:21.683676 9056 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:00:21.683674 9449 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
I20240501 08:00:21.684656 9318 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
I20240501 08:00:21.685359 10335 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:21.687028 9056 raft_consensus.cc:858] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }:
New config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
W20240501 08:00:21.687110 9056 raft_consensus.cc:1565] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 12' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }:
New config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
I20240501 08:00:21.687592 9056 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
W20240501 08:00:21.689657 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:21.691790 7171 catalog_manager.cc:5484] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776 reported cstate change: config changed from index 11 to 12, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
W20240501 08:00:21.693185 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:21.693250 7513 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:21.693284 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:21.700594 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5970772c02d04e0b95a2f2feb7dcd655: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:00:21.701514 7175 catalog_manager.cc:5484] T f647cfc562794e65a05918b241014d72 P 1807c0c2e7d04b20b0cda311c42eef8e reported cstate change: config changed from index 7 to 8, 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:00:21.706868 12302 tablet_bootstrap.cc:492] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:21.707202 12302 tablet_bootstrap.cc:492] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap complete.
I20240501 08:00:21.707399 12302 ts_tablet_manager.cc:1367] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.043s user 0.029s sys 0.004s
I20240501 08:00:21.707828 12302 raft_consensus.cc:348] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } }
I20240501 08:00:21.707947 12302 raft_consensus.cc:729] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: LEARNER
I20240501 08:00:21.708031 12302 consensus_queue.cc:260] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } }
I20240501 08:00:21.708387 12302 ts_tablet_manager.cc:1398] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 08:00:21.708945 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session 31b02f06be8a4455b94b1254d77005bd-e178f7c3c17c4e1ca3d16b6441f76e77 received from {username='slave'} at 127.0.99.131:52783
I20240501 08:00:21.709038 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session 31b02f06be8a4455b94b1254d77005bd-e178f7c3c17c4e1ca3d16b6441f76e77 on tablet e178f7c3c17c4e1ca3d16b6441f76e77 with peer 31b02f06be8a4455b94b1254d77005bd
W20240501 08:00:21.718384 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:21.718490 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:21.718528 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:21.718566 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:21.719614 11769 tablet_copy_client.cc:523] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:21.727312 11769 tablet_bootstrap.cc:492] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap starting.
@ 0x7ff1b3ba9b93 google::LogMessage::SendToLog() at ??:0
I20240501 08:00:21.730460 9501 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 08:00:15.144878 9507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:15.145222 9507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:15.150405 9507 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:00:16.091435 9507 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 08:00:16.654196 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15bea2064e1c414fb23182f6bc324bfd: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:16.654312 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:16.654434 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 61deeb642b0b4f06a2ef4b1a391fdcb9: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:16.654759 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba8a985906c04117a440386d3011b936: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:16.655175 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2427aea1b9a6436ca7bd1796a85634ff: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:16.655249 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15bea2064e1c414fb23182f6bc324bfd: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:16.655320 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:16.655369 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:16.655422 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ae84a8bf8ac745fb982412e9b194d7fc: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:16.655526 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba8a985906c04117a440386d3011b936: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:16.655617 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22d40ca8e4714c2495974e28872c5687: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:16.655674 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:16.655992 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ae84a8bf8ac745fb982412e9b194d7fc: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:16.656057 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7ddae9ad2184cb9bd097d68171c3fc4: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:16.656458 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:16.656540 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba8a985906c04117a440386d3011b936: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:16.656620 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22d40ca8e4714c2495974e28872c5687: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:16.656658 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2427aea1b9a6436ca7bd1796a85634ff: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:16.820096 9507 rebalancer_tool.cc:1074] tablet 21d76484f5dd499fb6fc3d534bffb779: '31b02f06be8a4455b94b1254d77005bd' -> '1c8cb9b1059a47cc822ca13a2b931776' move scheduled
I20240501 08:00:16.854826 9507 rebalancer_tool.cc:1074] tablet 81ccdd855b584d65860fd7a5777920a3: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '3b69c5d799e14e438df760453438aa75' move scheduled
I20240501 08:00:16.885399 9507 rebalancer_tool.cc:1074] tablet 1e44704a9b9c46cf954d8e6937983841: 'b4f8020266b5455b872a2973fc396578' -> 'd8aa5ac376a54e1f8417e5af11d28fa7' move scheduled
I20240501 08:00:16.964291 9507 rebalancer_tool.cc:1074] tablet cc6f7dcc930e4937865d37cb3c3bb3d2: '09a7b6e2731a474b80411efb62323604' -> '1807c0c2e7d04b20b0cda311c42eef8e' move scheduled
I20240501 08:00:17.103761 9507 rebalancer_tool.cc:1074] tablet 1088a116ccec4020aec562a69791a188: '31b02f06be8a4455b94b1254d77005bd' -> '1c8cb9b1059a47cc822ca13a2b931776' move scheduled
I20240501 08:00:17.142181 9507 rebalancer_tool.cc:1074] tablet bde42e485c4b49ce8acd694a25ba683a: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '3b69c5d799e14e438df760453438aa75' move scheduled
I20240501 08:00:17.202859 9507 rebalancer_tool.cc:1074] tablet 4dde9710a5eb4baa881d9dd04affe184: 'b4f8020266b5455b872a2973fc396578' -> '1807c0c2e7d04b20b0cda311c42eef8e' move scheduled
I20240501 08:00:17.300763 9507 rebalancer_tool.cc:1074] tablet 02b42b5fd3d440ea950b35ed461b8e12: '09a7b6e2731a474b80411efb62323604' -> 'd8aa5ac376a54e1f8417e5af11d28fa7' move scheduled
I20240501 08:00:17.577335 9507 rebalancer_tool.cc:1074] tablet df6dd3d8482d4931b1d3014a67e739c6: '31b02f06be8a4455b94b1254d77005bd' -> '1c8cb9b1059a47cc822ca13a2b931776' move scheduled
I20240501 08:00:17.597767 9507 rebalancer_tool.cc:1084] tablet 5a37e631ae4f46a2be1ad5c43db5724b: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '3b69c5d799e14e438df760453438aa75' move ignored: Illegal state: tablet 5a37e631ae4f46a2be1ad5c43db5724b: replica 3b69c5d799e14e438df760453438aa75 is already present
I20240501 08:00:17.597895 9507 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:00:18.232733 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:18.245870 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7ddae9ad2184cb9bd097d68171c3fc4: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:18.246040 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:18.246896 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 61deeb642b0b4f06a2ef4b1a391fdcb9: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:18.247028 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22d40ca8e4714c2495974e28872c5687: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:18.247082 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:18.248010 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ae84a8bf8ac745fb982412e9b194d7fc: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:18.248102 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:18.248157 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:18.248211 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba8a985906c04117a440386d3011b936: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:18.248315 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2427aea1b9a6436ca7bd1796a85634ff: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:18.248379 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:18.249059 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 61deeb642b0b4f06a2ef4b1a391fdcb9: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:18.249142 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15bea2064e1c414fb23182f6bc324bfd: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:18.249200 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:18.330458 9507 rebalancer_tool.cc:1074] tablet ffc23756b44d4d588e0a8175075d9bc6: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '1807c0c2e7d04b20b0cda311c42eef8e' move scheduled
I20240501 08:00:18.470901 9507 rebalancer_tool.cc:1074] tablet 4ea0db42a75a4879b3d3d608666dc2bf: 'b4f8020266b5455b872a2973fc396578' -> '1807c0c2e7d04b20b0cda311c42eef8e' move scheduled
I20240501 08:00:18.620262 9507 rebalancer_tool.cc:1084] tablet 07967f68a412411494c6dd1f47209009: '09a7b6e2731a474b80411efb62323604' -> '3b69c5d799e14e438df760453438aa75' move ignored: Illegal state: tablet 07967f68a412411494c6dd1f47209009: replica 3b69c5d799e14e438df760453438aa75 is already present
I20240501 08:00:18.620373 9507 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:00:19.406163 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 09a7b6e2731a474b80411efb62323604
W20240501 08:00:19.406253 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2427aea1b9a6436ca7bd1796a85634ff: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:19.406338 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:19.406443 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:19.406539 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15bea2064e1c414fb23182f6bc324bfd: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:19.406580 9507 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:19.473879 9507 rebalancer_tool.cc:1074] tablet 5970772c02d04e0b95a2f2feb7dcd655: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '3b69c5d799e14e438df760453438aa75' move scheduled
I20240501 08:00:19.522732 9507 rebalancer_tool.cc:1074] tablet 8882528b26974c7aa73bb0ff0c444836: 'b4f8020266b5455b872a2973fc396578' -> 'd8aa5ac376a54e1f8417e5af11d28fa7' move scheduled
I20240501 08:00:19.661867 9507 rebalancer_tool.cc:1074] tablet b357c740aca144b7b063845726f419e2: '09a7b6e2731a474b80411efb62323604' -> '1807c0c2e7d04b20b0cda311c42eef8e' move scheduled
I20240501 08:00:19.863193 9507 rebalancer_tool.cc:1074] tablet 7377810e08104526b8b8acdf886a3368: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '1c8cb9b1059a47cc822ca13a2b931776' move scheduled
I20240501 08:00:19.919880 9507 rebalancer_tool.cc:1074] tablet 28baea6d7d3f442ba1dbfc5b37de90ac: 'b4f8020266b5455b872a2973fc396578' -> '3b69c5d799e14e438df760453438aa75' move scheduled
I20240501 08:00:20.200505 9507 rebalancer_tool.cc:1136] tablet df6dd3d8482d4931b1d3014a67e739c6: '31b02f06be8a4455b94b1254d77005bd' -> '1c8cb9b1059a47cc822ca13a2b931776' move completed: OK
I20240501 08:00:20.316099 9507 rebalancer_tool.cc:1136] tablet 02b42b5fd3d440ea950b35ed461b8e12: '09a7b6e2731a474b80411efb62323604' -> 'd8aa5ac376a54e1f8417e5af11d28fa7' move completed: OK
I20240501 08:00:20.403746 9507 rebalancer_tool.cc:1136] tablet 4dde9710a5eb4baa881d9dd04affe184: 'b4f8020266b5455b872a2973fc396578' -> '1807c0c2e7d04b20b0cda311c42eef8e' move completed: OK
I20240501 08:00:20.450083 9507 rebalancer_tool.cc:1136] tablet cc6f7dcc930e4937865d37cb3c3bb3d2: '09a7b6e2731a474b80411efb62323604' -> '1807c0c2e7d04b20b0cda311c42eef8e' move completed: OK
I20240501 08:00:20.519843 9507 rebalancer_tool.cc:1136] tablet 21d76484f5dd499fb6fc3d534bffb779: '31b02f06be8a4455b94b1254d77005bd' -> '1c8cb9b1059a47cc822ca13a2b931776' move completed: Incomplete: tablet 21d76484f5dd499fb6fc3d534bffb779, TS 31b02f06be8a4455b94b1254d77005bd -> TS 1c8cb9b1059a47cc822ca13a2b931776 move failed, target replica disappeared
I20240501 08:00:20.562986 9507 rebalancer_tool.cc:1136] tablet 1e44704a9b9c46cf954d8e6937983841: 'b4f8020266b5455b872a2973fc396578' -> 'd8aa5ac376a54e1f8417e5af11d28fa7' move completed: OK
I20240501 08:00:20.646589 9507 rebalancer_tool.cc:1136] tablet 81ccdd855b584d65860fd7a5777920a3: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '3b69c5d799e14e438df760453438aa75' move completed: OK
I20240501 08:00:20.666148 9507 rebalancer_tool.cc:1136] tablet 1088a116ccec4020aec562a69791a188: '31b02f06be8a4455b94b1254d77005bd' -> '1c8cb9b1059a47cc822ca13a2b931776' move completed: OK
I20240501 08:00:20.764564 9507 rebalancer_tool.cc:1136] tablet bde42e485c4b49ce8acd694a25ba683a: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '3b69c5d799e14e438df760453438aa75' move completed: Incomplete: tablet bde42e485c4b49ce8acd694a25ba683a, TS d6b4be7bbd1649d0b7c5e8efe81f759f -> TS 3b69c5d799e14e438df760453438aa75 move failed, target replica disappeared
I20240501 08:00:20.816632 9507 rebalancer_tool.cc:1084] tablet f647cfc562794e65a05918b241014d72: '31b02f06be8a4455b94b1254d77005bd' -> '1807c0c2e7d04b20b0cda311c42eef8e' move ignored: Illegal state: tablet f647cfc562794e65a05918b241014d72: replica 1807c0c2e7d04b20b0cda311c42eef8e is already present
I20240501 08:00:20.816720 9507 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:00:21.628708 12000 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:00:21.267790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.134:33589 (local address 127.0.0.1:57108)
0501 08:00:21.273885 (+ 6095us) negotiation.cc:107] Waiting for socket to connect
0501 08:00:21.273898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 08:00:21.273958 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:00:21.627963 (+354005us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":6046,"thread_start_us":48,"threads_started":1}
Illegal state: tablet server 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): unacceptable health status UNAVAILABLE
@ 0x7ff1b3ba57cc google::LogMessage::Flush() at ??:0
@ 0x7ff1b3ba6f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 08:00:21.740952 9056 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
I20240501 08:00:21.741765 7707 consensus_queue.cc:237] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
I20240501 08:00:21.744235 9449 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20240501 08:00:21.746239 7646 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): Couldn't send request to peer b4f8020266b5455b872a2973fc396578. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
@ 0x7ff1b500f044 kudu::TraceMetrics::InternName() at ??:0
I20240501 08:00:21.747457 12248 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:00:21.750232 10335 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
I20240501 08:00:21.752108 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4dde9710a5eb4baa881d9dd04affe184 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:21.753094 9449 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
I20240501 08:00:21.756309 7171 catalog_manager.cc:5484] T 4dde9710a5eb4baa881d9dd04affe184 P 1807c0c2e7d04b20b0cda311c42eef8e reported cstate change: config changed from index 12 to 13, VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
I20240501 08:00:21.756922 9056 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 08:00:21.757504 12248 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:00:21.758481 9056 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
I20240501 08:00:21.762899 9298 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dde9710a5eb4baa881d9dd04affe184 with delete_type TABLET_DATA_TOMBSTONED (TS 1c8cb9b1059a47cc822ca13a2b931776 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:40266
I20240501 08:00:21.771822 11769 tablet_bootstrap.cc:492] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:21.772172 11769 tablet_bootstrap.cc:492] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776: Bootstrap complete.
I20240501 08:00:21.772364 11769 ts_tablet_manager.cc:1367] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776: Time spent bootstrapping tablet: real 0.045s user 0.030s sys 0.001s
@ 0x7ff1b4ff3f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240501 08:00:21.774240 11769 raft_consensus.cc:348] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:21.774372 11769 raft_consensus.cc:729] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c8cb9b1059a47cc822ca13a2b931776, State: Initialized, Role: LEARNER
I20240501 08:00:21.774478 11769 consensus_queue.cc:260] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:00:21.775758 12361 tablet_replica.cc:330] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776: stopping tablet replica
I20240501 08:00:21.775889 12361 raft_consensus.cc:2229] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:21.777179 11769 ts_tablet_manager.cc:1398] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:00:21.778079 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-28baea6d7d3f442ba1dbfc5b37de90ac received from {username='slave'} at 127.0.99.135:36909
I20240501 08:00:21.778077 7313 consensus_queue.cc:237] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } }
I20240501 08:00:21.778177 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session 1c8cb9b1059a47cc822ca13a2b931776-28baea6d7d3f442ba1dbfc5b37de90ac on tablet 28baea6d7d3f442ba1dbfc5b37de90ac with peer 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:21.780225 7707 raft_consensus.cc:1261] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:21.781149 12104 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:21.781589 12361 raft_consensus.cc:2258] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 08:00:21.781653 7250 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 -> Peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861): Couldn't send request to peer 31b02f06be8a4455b94b1254d77005bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8fdc2b9c2ef44b0bfec59a1a2f6b626. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:21.781924 7251 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:21.783172 12361 ts_tablet_manager.cc:1875] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:21.787195 12361 ts_tablet_manager.cc:1888] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240501 08:00:21.787266 12361 log.cc:1196] T 4dde9710a5eb4baa881d9dd04affe184 P 1c8cb9b1059a47cc822ca13a2b931776: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/4dde9710a5eb4baa881d9dd04affe184
I20240501 08:00:21.788064 7159 catalog_manager.cc:4830] TS 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): tablet 4dde9710a5eb4baa881d9dd04affe184 (table rebalance_test_table_4 [id=ae84a8bf8ac745fb982412e9b194d7fc]) successfully deleted
@ 0x7ff1b4ff1efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 08:00:21.790354 9502 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 08:00:15.091753 9506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:15.092136 9506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:15.097623 9506 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:00:16.048148 9506 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 08:00:16.509241 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:16.509423 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:16.509526 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15bea2064e1c414fb23182f6bc324bfd: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:16.509559 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2427aea1b9a6436ca7bd1796a85634ff: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:16.509613 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ae84a8bf8ac745fb982412e9b194d7fc: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:16.509675 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:16.509717 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba8a985906c04117a440386d3011b936: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:16.509747 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:16.509774 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15bea2064e1c414fb23182f6bc324bfd: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:16.509819 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22d40ca8e4714c2495974e28872c5687: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:16.509850 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 61deeb642b0b4f06a2ef4b1a391fdcb9: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:16.509896 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:16.509943 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7ddae9ad2184cb9bd097d68171c3fc4: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:16.509975 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba8a985906c04117a440386d3011b936: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:16.510061 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2427aea1b9a6436ca7bd1796a85634ff: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:16.610659 9506 rebalancer_tool.cc:1074] tablet 3abb5202d8d24d9b9a67f7a87b7300a6: 'b4f8020266b5455b872a2973fc396578' -> '1c8cb9b1059a47cc822ca13a2b931776' move scheduled
I20240501 08:00:16.636348 9506 rebalancer_tool.cc:1084] tablet 54ced80b9ff943ec88c27b7842500b3d: '31b02f06be8a4455b94b1254d77005bd' -> 'd8aa5ac376a54e1f8417e5af11d28fa7' move ignored: Illegal state: tablet 54ced80b9ff943ec88c27b7842500b3d: replica d8aa5ac376a54e1f8417e5af11d28fa7 is already present
I20240501 08:00:16.636462 9506 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:00:17.118799 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ae84a8bf8ac745fb982412e9b194d7fc: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:17.118875 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:17.118958 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15bea2064e1c414fb23182f6bc324bfd: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:17.119007 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:17.119051 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ae84a8bf8ac745fb982412e9b194d7fc: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:17.119122 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7ddae9ad2184cb9bd097d68171c3fc4: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:17.119190 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22d40ca8e4714c2495974e28872c5687: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:17.119249 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:17.119292 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7ddae9ad2184cb9bd097d68171c3fc4: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:17.119341 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2427aea1b9a6436ca7bd1796a85634ff: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:17.119385 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba8a985906c04117a440386d3011b936: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:17.119426 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15bea2064e1c414fb23182f6bc324bfd: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:17.119465 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:17.119508 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 61deeb642b0b4f06a2ef4b1a391fdcb9: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:17.119547 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ae84a8bf8ac745fb982412e9b194d7fc: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:17.169200 9506 rebalancer_tool.cc:1074] tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '3b69c5d799e14e438df760453438aa75' move scheduled
I20240501 08:00:17.195171 9506 rebalancer_tool.cc:1074] tablet 21d76484f5dd499fb6fc3d534bffb779: '31b02f06be8a4455b94b1254d77005bd' -> '1807c0c2e7d04b20b0cda311c42eef8e' move scheduled
I20240501 08:00:17.264251 9506 rebalancer_tool.cc:1074] tablet 8311d0921e9e417090e441884615525a: 'b4f8020266b5455b872a2973fc396578' -> 'd8aa5ac376a54e1f8417e5af11d28fa7' move scheduled
I20240501 08:00:17.322757 9506 rebalancer_tool.cc:1074] tablet 0e42ef8df4884bc9bc33f5ed4c164c8c: '09a7b6e2731a474b80411efb62323604' -> '1c8cb9b1059a47cc822ca13a2b931776' move scheduled
I20240501 08:00:17.421452 9506 rebalancer_tool.cc:1074] tablet 5a37e631ae4f46a2be1ad5c43db5724b: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '3b69c5d799e14e438df760453438aa75' move scheduled
I20240501 08:00:17.591197 9506 rebalancer_tool.cc:1074] tablet dd17760cd5c2454db97610b2f71c0ee2: '31b02f06be8a4455b94b1254d77005bd' -> '1807c0c2e7d04b20b0cda311c42eef8e' move scheduled
I20240501 08:00:17.632647 9506 rebalancer_tool.cc:1074] tablet b357c740aca144b7b063845726f419e2: 'b4f8020266b5455b872a2973fc396578' -> 'd8aa5ac376a54e1f8417e5af11d28fa7' move scheduled
I20240501 08:00:17.693195 9506 rebalancer_tool.cc:1074] tablet 02b42b5fd3d440ea950b35ed461b8e12: '09a7b6e2731a474b80411efb62323604' -> '3b69c5d799e14e438df760453438aa75' move scheduled
I20240501 08:00:17.882357 9506 rebalancer_tool.cc:1084] tablet 81ccdd855b584d65860fd7a5777920a3: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '1c8cb9b1059a47cc822ca13a2b931776' move ignored: Illegal state: tablet 81ccdd855b584d65860fd7a5777920a3: replica 1c8cb9b1059a47cc822ca13a2b931776 is already present
I20240501 08:00:17.882443 9506 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:00:18.643935 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:18.650944 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:18.651119 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba8a985906c04117a440386d3011b936: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:18.651222 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22d40ca8e4714c2495974e28872c5687: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:18.651321 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ae84a8bf8ac745fb982412e9b194d7fc: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:18.651405 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:18.651459 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:18.651552 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22d40ca8e4714c2495974e28872c5687: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:18.651607 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ae84a8bf8ac745fb982412e9b194d7fc: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:18.651656 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:18.651703 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 61deeb642b0b4f06a2ef4b1a391fdcb9: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:18.652518 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2427aea1b9a6436ca7bd1796a85634ff: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:18.652595 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:18.652648 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7ddae9ad2184cb9bd097d68171c3fc4: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:18.809851 9506 rebalancer_tool.cc:1084] tablet 1e44704a9b9c46cf954d8e6937983841: '31b02f06be8a4455b94b1254d77005bd' -> '1c8cb9b1059a47cc822ca13a2b931776' move ignored: Not found: Peer with uuid 31b02f06be8a4455b94b1254d77005bd not found in consensus config
I20240501 08:00:18.809949 9506 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:00:19.913537 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 61deeb642b0b4f06a2ef4b1a391fdcb9: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:19.913674 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:19.913779 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2427aea1b9a6436ca7bd1796a85634ff: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 3b69c5d799e14e438df760453438aa75
I20240501 08:00:19.951648 9506 rebalancer_tool.cc:1074] tablet 4dde9710a5eb4baa881d9dd04affe184: '09a7b6e2731a474b80411efb62323604' -> '1c8cb9b1059a47cc822ca13a2b931776' move scheduled
I20240501 08:00:20.005321 9506 rebalancer_tool.cc:1074] tablet 5970772c02d04e0b95a2f2feb7dcd655: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '09a7b6e2731a474b80411efb62323604' move scheduled
I20240501 08:00:20.113229 9506 rebalancer_tool.cc:1074] tablet f647cfc562794e65a05918b241014d72: '31b02f06be8a4455b94b1254d77005bd' -> '1807c0c2e7d04b20b0cda311c42eef8e' move scheduled
I20240501 08:00:20.273319 9506 rebalancer_tool.cc:1074] tablet df6dd3d8482d4931b1d3014a67e739c6: 'b4f8020266b5455b872a2973fc396578' -> 'd8aa5ac376a54e1f8417e5af11d28fa7' move scheduled
I20240501 08:00:20.455844 9506 rebalancer_tool.cc:1084] tablet ffc23756b44d4d588e0a8175075d9bc6: '09a7b6e2731a474b80411efb62323604' -> '3b69c5d799e14e438df760453438aa75' move ignored: Illegal state: Replica d6b4be7bbd1649d0b7c5e8efe81f759f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:00:20.455930 9506 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:00:21.167526 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22d40ca8e4714c2495974e28872c5687: could not find any suitable replica to move from server d8aa5ac376a54e1f8417e5af11d28fa7 to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:21.167706 9506 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ae84a8bf8ac745fb982412e9b194d7fc: could not find any suitable replica to move from server 1c8cb9b1059a47cc822ca13a2b931776 to server b4f8020266b5455b872a2973fc396578
I20240501 08:00:21.229735 9506 rebalancer_tool.cc:1074] tablet e178f7c3c17c4e1ca3d16b6441f76e77: '1c8cb9b1059a47cc822ca13a2b931776' -> '31b02f06be8a4455b94b1254d77005bd' move scheduled
I20240501 08:00:21.264740 9506 rebalancer_tool.cc:1084] tablet 1088a116ccec4020aec562a69791a188: 'b4f8020266b5455b872a2973fc396578' -> '1807c0c2e7d04b20b0cda311c42eef8e' move ignored: Illegal state: tablet 1088a116ccec4020aec562a69791a188: replica 1807c0c2e7d04b20b0cda311c42eef8e is already present
I20240501 08:00:21.264828 9506 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:00:21.628701 12243 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:00:21.498587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.134:33589 (local address 127.0.0.1:57114)
0501 08:00:21.500046 (+ 1459us) negotiation.cc:107] Waiting for socket to connect
0501 08:00:21.500059 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 08:00:21.500138 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:00:21.627931 (+127793us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1373}
Illegal state: tablet server 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): unacceptable health status UNAVAILABLE
@ 0x7ff1b7ba477a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 08:00:21.795459 12104 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } } }
@ 0x7ff1b7bada45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 08:00:21.798365 7707 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } } }
I20240501 08:00:21.798854 7167 catalog_manager.cc:5484] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 6 to 7, NON_VOTER 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:21.801107 12383 ts_tablet_manager.cc:902] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:00:21.801894 12383 tablet_copy_client.cc:308] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
@ 0x7ff1b7c2e76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 08:00:21.802577 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet cc6f7dcc930e4937865d37cb3c3bb3d2 from peer 09a7b6e2731a474b80411efb62323604 ({username='slave'} at 127.0.99.132:55545)
I20240501 08:00:21.802680 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet cc6f7dcc930e4937865d37cb3c3bb3d2 from peer 09a7b6e2731a474b80411efb62323604 at {username='slave'} at 127.0.99.132:55545: session id = 09a7b6e2731a474b80411efb62323604-cc6f7dcc930e4937865d37cb3c3bb3d2
I20240501 08:00:21.803675 7599 tablet_copy_source_session.cc:205] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:21.805141 12383 ts_tablet_manager.cc:1875] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: Deleting tablet data with delete state TABLET_DATA_COPYING
@ 0x7ff1b7c424f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7ff1b7c2d226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240501 08:00:21.810168 12383 ts_tablet_manager.cc:1888] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240501 08:00:21.810393 12383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc6f7dcc930e4937865d37cb3c3bb3d2. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7ff1b7c3cc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240501 08:00:21.813501 12383 tablet_copy_client.cc:783] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:21.813699 12383 tablet_copy_client.cc:647] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: tablet copy: Starting download of 1 WAL segments...
@ 0x556d501c175c std::function<>::operator()() at ??:0
I20240501 08:00:21.815100 12383 tablet_copy_client.cc:523] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:21.816799 12383 tablet_bootstrap.cc:492] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: Bootstrap starting.
@ 0x7ff1b4ff6e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff1b4ff76ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240501 08:00:21.843395 10907 raft_consensus.cc:1050] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 to VOTER
I20240501 08:00:21.843806 10907 consensus_queue.cc:237] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:21.844573 11072 raft_consensus.cc:1050] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 to VOTER
I20240501 08:00:21.844969 11072 consensus_queue.cc:237] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:21.845510 7707 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 08:00:21.846012 11880 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240501 08:00:21.846393 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:21.846480 7313 raft_consensus.cc:1261] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:21.846765 9318 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 08:00:21.846832 9056 raft_consensus.cc:1261] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:21.846889 12248 consensus_queue.cc:1035] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:21.847186 12248 consensus_queue.cc:1035] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:21.848012 11880 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 08:00:21.848551 9318 raft_consensus.cc:1261] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:21.848999 12248 consensus_queue.cc:1035] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
@ 0x7ff1b4ff903b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x556d501c175c std::function<>::operator()() at ??:0
I20240501 08:00:21.853103 12248 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:21.854457 7313 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:21.857326 7172 catalog_manager.cc:5484] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 7 to 8, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:21.859081 9056 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:21.859184 9319 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:21.863916 12383 tablet_bootstrap.cc:492] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:21.864269 12383 tablet_bootstrap.cc:492] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: Bootstrap complete.
I20240501 08:00:21.864687 12383 ts_tablet_manager.cc:1367] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: Time spent bootstrapping tablet: real 0.048s user 0.030s sys 0.001s
I20240501 08:00:21.865514 7707 consensus_queue.cc:237] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:21.867130 9318 raft_consensus.cc:1261] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:21.867295 9056 raft_consensus.cc:1261] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:21.867483 12383 raft_consensus.cc:348] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: true } }
I20240501 08:00:21.867544 11072 consensus_queue.cc:1035] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:21.867604 12383 raft_consensus.cc:729] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Initialized, Role: LEARNER
I20240501 08:00:21.867693 12383 consensus_queue.cc:260] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: true } }
I20240501 08:00:21.867877 12248 consensus_queue.cc:1035] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:21.869087 7313 consensus_queue.cc:237] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } }
I20240501 08:00:21.870455 12383 ts_tablet_manager.cc:1398] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:00:21.870939 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session 09a7b6e2731a474b80411efb62323604-cc6f7dcc930e4937865d37cb3c3bb3d2 received from {username='slave'} at 127.0.99.132:55545
I20240501 08:00:21.871016 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session 09a7b6e2731a474b80411efb62323604-cc6f7dcc930e4937865d37cb3c3bb3d2 on tablet cc6f7dcc930e4937865d37cb3c3bb3d2 with peer 09a7b6e2731a474b80411efb62323604
I20240501 08:00:21.870889 12247 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:21.872735 9318 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:21.872920 11880 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
@ 0x7ff1b4fe6807 kudu::Thread::SuperviseThread() at ??:0
I20240501 08:00:21.874358 9319 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:21.875304 7707 raft_consensus.cc:1261] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:21.875702 11143 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:21.877781 7175 catalog_manager.cc:5484] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 14 to 15, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:21.877967 9319 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:21.878233 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df6dd3d8482d4931b1d3014a67e739c6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:21.878841 7709 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:21.878978 7171 catalog_manager.cc:5484] T df6dd3d8482d4931b1d3014a67e739c6 P 1c8cb9b1059a47cc822ca13a2b931776 reported cstate change: config changed from index 8 to 9, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:21.879438 11143 consensus_queue.cc:1035] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
@ 0x7ff1b58246db start_thread at ??:0
I20240501 08:00:21.882671 11143 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
I20240501 08:00:21.883272 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7ff1b2d5c71f clone at ??:0
I20240501 08:00:21.885807 7297 tablet_service.cc:1514] Processing DeleteTablet for tablet df6dd3d8482d4931b1d3014a67e739c6 with delete_type TABLET_DATA_TOMBSTONED (TS b4f8020266b5455b872a2973fc396578 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35594
I20240501 08:00:21.886068 7172 catalog_manager.cc:5484] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 10 to 11, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:21.886255 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5970772c02d04e0b95a2f2feb7dcd655 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:21.888243 7707 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
I20240501 08:00:21.891160 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet 5970772c02d04e0b95a2f2feb7dcd655 with delete_type TABLET_DATA_TOMBSTONED (TS d6b4be7bbd1649d0b7c5e8efe81f759f not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:45222
I20240501 08:00:21.891530 12425 tablet_replica.cc:330] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f: stopping tablet replica
I20240501 08:00:21.891677 12425 raft_consensus.cc:2229] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:21.891952 12425 raft_consensus.cc:2258] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:21.891983 9319 raft_consensus.cc:2932] T 5970772c02d04e0b95a2f2feb7dcd655 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
I20240501 08:00:21.898913 12425 ts_tablet_manager.cc:1875] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:21.901400 12425 ts_tablet_manager.cc:1888] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 08:00:21.901470 12425 log.cc:1196] T 5970772c02d04e0b95a2f2feb7dcd655 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5970772c02d04e0b95a2f2feb7dcd655
I20240501 08:00:21.902065 7159 catalog_manager.cc:4830] TS d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707): tablet 5970772c02d04e0b95a2f2feb7dcd655 (table rebalance_test_table_2 [id=d578050a40e0474484896c98e4da9f5a]) successfully deleted
I20240501 08:00:21.902968 9056 raft_consensus.cc:2932] T df6dd3d8482d4931b1d3014a67e739c6 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:21.905309 9319 raft_consensus.cc:1203] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:00:21.911490 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:21.916756 7428 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:21.936924 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:21.937575 12424 tablet_replica.cc:330] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578: stopping tablet replica
I20240501 08:00:21.937748 12424 raft_consensus.cc:2229] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:21.938445 12424 raft_consensus.cc:2258] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:21.939744 12424 ts_tablet_manager.cc:1875] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:21.946491 12424 ts_tablet_manager.cc:1888] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:00:21.946575 12424 log.cc:1196] T df6dd3d8482d4931b1d3014a67e739c6 P b4f8020266b5455b872a2973fc396578: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/df6dd3d8482d4931b1d3014a67e739c6
I20240501 08:00:21.947240 7162 catalog_manager.cc:4830] TS b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): tablet df6dd3d8482d4931b1d3014a67e739c6 (table rebalance_test_table_7 [id=c7ddae9ad2184cb9bd097d68171c3fc4]) successfully deleted
I20240501 08:00:21.950896 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:21.953212 7688 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:22.014531 7575 raft_consensus.cc:1203] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 08:00:22.017243 11088 raft_consensus.cc:1050] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 to VOTER
I20240501 08:00:22.017755 11088 consensus_queue.cc:237] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:00:22.019258 7575 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:22.019519 9056 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:22.019604 10335 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:22.019950 10335 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:00:22.024211 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:22.029814 10335 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:22.032378 9298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:22.033627 9056 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:22.033695 7169 catalog_manager.cc:5484] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 7 to 8, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:22.034083 7688 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:22.037331 7575 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:00:22.044625 7558 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:22.054165 7297 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:22.059454 9037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:22.095702 7428 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:22.119524 7575 consensus_queue.cc:237] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:22.120754 9319 raft_consensus.cc:1261] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:22.121107 10907 consensus_queue.cc:1035] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:22.122570 11851 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:22.123013 9319 raft_consensus.cc:2932] T 7377810e08104526b8b8acdf886a3368 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:22.125496 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7377810e08104526b8b8acdf886a3368 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:22.125558 7169 catalog_manager.cc:5484] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 8 to 9, VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:22.128312 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet 7377810e08104526b8b8acdf886a3368 with delete_type TABLET_DATA_TOMBSTONED (TS d6b4be7bbd1649d0b7c5e8efe81f759f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45222
I20240501 08:00:22.128554 12425 tablet_replica.cc:330] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f: stopping tablet replica
I20240501 08:00:22.128669 12425 raft_consensus.cc:2229] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:22.128830 12425 raft_consensus.cc:2258] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:22.129539 12425 ts_tablet_manager.cc:1875] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:22.131377 12425 ts_tablet_manager.cc:1888] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:00:22.131451 12425 log.cc:1196] T 7377810e08104526b8b8acdf886a3368 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7377810e08104526b8b8acdf886a3368
I20240501 08:00:22.131964 7159 catalog_manager.cc:4830] TS d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707): tablet 7377810e08104526b8b8acdf886a3368 (table rebalance_test_table_4 [id=ae84a8bf8ac745fb982412e9b194d7fc]) successfully deleted
I20240501 08:00:22.157788 12302 ts_tablet_manager.cc:902] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: Initiating tablet copy from peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:00:22.158434 12302 tablet_copy_client.cc:308] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40707
I20240501 08:00:22.159021 7467 tablet_copy_service.cc:140] P d6b4be7bbd1649d0b7c5e8efe81f759f: Received BeginTabletCopySession request for tablet dd17760cd5c2454db97610b2f71c0ee2 from peer 31b02f06be8a4455b94b1254d77005bd ({username='slave'} at 127.0.99.131:53791)
I20240501 08:00:22.159129 7467 tablet_copy_service.cc:161] P d6b4be7bbd1649d0b7c5e8efe81f759f: Beginning new tablet copy session on tablet dd17760cd5c2454db97610b2f71c0ee2 from peer 31b02f06be8a4455b94b1254d77005bd at {username='slave'} at 127.0.99.131:53791: session id = 31b02f06be8a4455b94b1254d77005bd-dd17760cd5c2454db97610b2f71c0ee2
I20240501 08:00:22.160004 7467 tablet_copy_source_session.cc:205] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:22.160785 12302 ts_tablet_manager.cc:1875] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240501 08:00:22.162868 7382 connection.cc:535] client connection to 127.0.99.136:42587 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:00:22.163607 12302 ts_tablet_manager.cc:1888] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 08:00:22.163789 12302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd17760cd5c2454db97610b2f71c0ee2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:00:22.163843 7644 connection.cc:535] client connection to 127.0.99.136:42587 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:22.164585 7251 connection.cc:535] client connection to 127.0.99.136:42587 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:22.164878 7513 connection.cc:535] client connection to 127.0.99.136:42587 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:00:22.164902 12302 tablet_copy_client.cc:783] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:22.165046 12302 tablet_copy_client.cc:647] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: tablet copy: Starting download of 1 WAL segments...
W20240501 08:00:22.165096 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:22.165182 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:22.166148 12315 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:00:22.163453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:42587 (local address 127.0.99.130:58767)
0501 08:00:22.163638 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0501 08:00:22.163655 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0501 08:00:22.163760 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:00:22.165762 (+ 2002us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":100}
W20240501 08:00:22.166481 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:22.167696 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:22.167775 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:22.168051 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:22.168123 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:22.168176 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:22.171970 12302 tablet_copy_client.cc:523] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:22.173586 12302 tablet_bootstrap.cc:492] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
I20240501 08:00:22.186189 12500 ts_tablet_manager.cc:902] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:00:22.186740 12500 tablet_copy_client.cc:308] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:00:22.187228 7728 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 4dde9710a5eb4baa881d9dd04affe184 from peer b4f8020266b5455b872a2973fc396578 ({username='slave'} at 127.0.99.129:43719)
I20240501 08:00:22.187330 7728 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 4dde9710a5eb4baa881d9dd04affe184 from peer b4f8020266b5455b872a2973fc396578 at {username='slave'} at 127.0.99.129:43719: session id = b4f8020266b5455b872a2973fc396578-4dde9710a5eb4baa881d9dd04affe184
I20240501 08:00:22.188162 7728 tablet_copy_source_session.cc:205] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:22.188827 12500 ts_tablet_manager.cc:1875] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:00:22.189357 9503 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 08:00:15.080510 9504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:15.080835 9504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:15.085189 9504 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:00:16.013494 9504 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 08:00:16.424530 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba8a985906c04117a440386d3011b936: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:16.424712 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2427aea1b9a6436ca7bd1796a85634ff: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:16.424786 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:16.424919 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba8a985906c04117a440386d3011b936: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:16.424974 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15bea2064e1c414fb23182f6bc324bfd: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:16.425061 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22d40ca8e4714c2495974e28872c5687: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:16.425112 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ae84a8bf8ac745fb982412e9b194d7fc: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:16.425165 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7ddae9ad2184cb9bd097d68171c3fc4: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:16.425212 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:16.425258 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:16.425326 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 61deeb642b0b4f06a2ef4b1a391fdcb9: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:16.425392 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15bea2064e1c414fb23182f6bc324bfd: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:16.425441 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2427aea1b9a6436ca7bd1796a85634ff: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:16.425508 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:16.460779 9504 rebalancer_tool.cc:1074] tablet 54ced80b9ff943ec88c27b7842500b3d: '31b02f06be8a4455b94b1254d77005bd' -> 'd8aa5ac376a54e1f8417e5af11d28fa7' move scheduled
I20240501 08:00:16.518553 9504 rebalancer_tool.cc:1074] tablet 7377810e08104526b8b8acdf886a3368: 'b4f8020266b5455b872a2973fc396578' -> '1807c0c2e7d04b20b0cda311c42eef8e' move scheduled
I20240501 08:00:16.580396 9504 rebalancer_tool.cc:1074] tablet 21d76484f5dd499fb6fc3d534bffb779: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '3b69c5d799e14e438df760453438aa75' move scheduled
I20240501 08:00:16.662215 9504 rebalancer_tool.cc:1074] tablet 5970772c02d04e0b95a2f2feb7dcd655: '09a7b6e2731a474b80411efb62323604' -> '1c8cb9b1059a47cc822ca13a2b931776' move scheduled
I20240501 08:00:16.781185 9504 rebalancer_tool.cc:1074] tablet e178f7c3c17c4e1ca3d16b6441f76e77: '31b02f06be8a4455b94b1254d77005bd' -> 'd8aa5ac376a54e1f8417e5af11d28fa7' move scheduled
I20240501 08:00:16.817049 9504 rebalancer_tool.cc:1074] tablet 3abb5202d8d24d9b9a67f7a87b7300a6: 'b4f8020266b5455b872a2973fc396578' -> '1807c0c2e7d04b20b0cda311c42eef8e' move scheduled
I20240501 08:00:16.864805 9504 rebalancer_tool.cc:1074] tablet bde42e485c4b49ce8acd694a25ba683a: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '1c8cb9b1059a47cc822ca13a2b931776' move scheduled
I20240501 08:00:16.940306 9504 rebalancer_tool.cc:1074] tablet f647cfc562794e65a05918b241014d72: '09a7b6e2731a474b80411efb62323604' -> '3b69c5d799e14e438df760453438aa75' move scheduled
I20240501 08:00:16.987267 9504 rebalancer_tool.cc:1074] tablet 4dde9710a5eb4baa881d9dd04affe184: '31b02f06be8a4455b94b1254d77005bd' -> 'd8aa5ac376a54e1f8417e5af11d28fa7' move scheduled
I20240501 08:00:17.086582 9504 rebalancer_tool.cc:1084] tablet cc6f7dcc930e4937865d37cb3c3bb3d2: 'b4f8020266b5455b872a2973fc396578' -> '1807c0c2e7d04b20b0cda311c42eef8e' move ignored: Illegal state: tablet cc6f7dcc930e4937865d37cb3c3bb3d2: replica 1807c0c2e7d04b20b0cda311c42eef8e is already present
I20240501 08:00:17.086668 9504 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:00:17.831413 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7ddae9ad2184cb9bd097d68171c3fc4: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:17.831588 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ae84a8bf8ac745fb982412e9b194d7fc: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:17.831665 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:17.831714 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7ddae9ad2184cb9bd097d68171c3fc4: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:17.831782 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:17.831851 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2427aea1b9a6436ca7bd1796a85634ff: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:17.831914 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba8a985906c04117a440386d3011b936: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:17.831980 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15bea2064e1c414fb23182f6bc324bfd: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:17.832031 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:17.832082 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22d40ca8e4714c2495974e28872c5687: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 3b69c5d799e14e438df760453438aa75
I20240501 08:00:18.219661 9504 rebalancer_tool.cc:1074] tablet 4ea0db42a75a4879b3d3d608666dc2bf: 'b4f8020266b5455b872a2973fc396578' -> '3b69c5d799e14e438df760453438aa75' move scheduled
I20240501 08:00:18.547065 9504 rebalancer_tool.cc:1074] tablet 81ccdd855b584d65860fd7a5777920a3: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '1807c0c2e7d04b20b0cda311c42eef8e' move scheduled
I20240501 08:00:18.868369 9504 rebalancer_tool.cc:1074] tablet 8311d0921e9e417090e441884615525a: '09a7b6e2731a474b80411efb62323604' -> '1c8cb9b1059a47cc822ca13a2b931776' move scheduled
I20240501 08:00:19.052341 9504 rebalancer_tool.cc:1074] tablet 1088a116ccec4020aec562a69791a188: '31b02f06be8a4455b94b1254d77005bd' -> '1807c0c2e7d04b20b0cda311c42eef8e' move scheduled
I20240501 08:00:19.175796 9504 rebalancer_tool.cc:1074] tablet ffc23756b44d4d588e0a8175075d9bc6: 'b4f8020266b5455b872a2973fc396578' -> 'd8aa5ac376a54e1f8417e5af11d28fa7' move scheduled
I20240501 08:00:19.426676 9504 rebalancer_tool.cc:1074] tablet 4297f95243ac431099a8279c3d9e4b25: 'b4f8020266b5455b872a2973fc396578' -> '3b69c5d799e14e438df760453438aa75' move scheduled
I20240501 08:00:19.506369 9504 rebalancer_tool.cc:1084] tablet cd07a00b6817400f86bf8c32bd592d8b: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '1c8cb9b1059a47cc822ca13a2b931776' move ignored: Not found: Peer with uuid d6b4be7bbd1649d0b7c5e8efe81f759f not found in consensus config
I20240501 08:00:19.506474 9504 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:00:20.479769 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba8a985906c04117a440386d3011b936: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server b4f8020266b5455b872a2973fc396578
W20240501 08:00:20.480744 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2427aea1b9a6436ca7bd1796a85634ff: could not find any suitable replica to move from server 1c8cb9b1059a47cc822ca13a2b931776 to server 31b02f06be8a4455b94b1254d77005bd
W20240501 08:00:20.481297 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7ddae9ad2184cb9bd097d68171c3fc4: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:20.481371 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 09a7b6e2731a474b80411efb62323604
W20240501 08:00:20.481444 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ae84a8bf8ac745fb982412e9b194d7fc: could not find any suitable replica to move from server 1807c0c2e7d04b20b0cda311c42eef8e to server b4f8020266b5455b872a2973fc396578
I20240501 08:00:20.564167 9504 rebalancer_tool.cc:1084] tablet 5d18a2918c75467a9fe59979ce8936cd: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '3b69c5d799e14e438df760453438aa75' move ignored: Illegal state: tablet 5d18a2918c75467a9fe59979ce8936cd: replica 3b69c5d799e14e438df760453438aa75 is already present
I20240501 08:00:20.564244 9504 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:00:21.246567 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15bea2064e1c414fb23182f6bc324bfd: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:21.246682 9504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba8a985906c04117a440386d3011b936: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server d8aa5ac376a54e1f8417e5af11d28fa7
I20240501 08:00:21.429338 9504 rebalancer_tool.cc:1074] tablet cc6f7dcc930e4937865d37cb3c3bb3d2: 'd8aa5ac376a54e1f8417e5af11d28fa7' -> '09a7b6e2731a474b80411efb62323604' move scheduled
I20240501 08:00:21.669209 9504 rebalancer_tool.cc:1084] tablet dd17760cd5c2454db97610b2f71c0ee2: '09a7b6e2731a474b80411efb62323604' -> '31b02f06be8a4455b94b1254d77005bd' move ignored: Illegal state: Request specified cas_config_opid_index of 6 but the committed config has opid_index of 7
I20240501 08:00:21.669298 9504 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:00:22.163406 12418 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:00:21.864244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:42587 (local address 127.0.0.1:58872)
0501 08:00:21.868929 (+ 4685us) negotiation.cc:107] Waiting for socket to connect
0501 08:00:21.868942 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 08:00:21.869578 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:00:21.929539 (+ 59961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 08:00:21.929542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 08:00:21.929565 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0501 08:00:21.929900 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 08:00:21.929911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 08:00:22.162554 (+232643us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":4618,"thread_start_us":65,"threads_started":1}
Illegal state: tablet server 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): unacceptable health status UNAVAILABLE
I20240501 08:00:22.191102 12500 ts_tablet_manager.cc:1888] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240501 08:00:22.191267 12500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dde9710a5eb4baa881d9dd04affe184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:22.192274 12500 tablet_copy_client.cc:783] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:22.192469 12500 tablet_copy_client.cc:647] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:22.193521 12500 tablet_copy_client.cc:523] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:22.195139 12500 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
W20240501 08:00:22.195626 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:00:22.203521 9499 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskS8gUPX/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 08:00:15.185839 9505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:00:15.186254 9505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:00:15.190339 9505 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:00:16.314988 9505 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 08:00:16.856684 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2427aea1b9a6436ca7bd1796a85634ff: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:16.856808 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:16.856902 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15bea2064e1c414fb23182f6bc324bfd: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:16.857152 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:16.857295 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:16.857360 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:16.857403 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2427aea1b9a6436ca7bd1796a85634ff: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:16.857445 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22d40ca8e4714c2495974e28872c5687: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:16.857499 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7ddae9ad2184cb9bd097d68171c3fc4: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:16.857568 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ae84a8bf8ac745fb982412e9b194d7fc: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:16.857611 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 61deeb642b0b4f06a2ef4b1a391fdcb9: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:16.857650 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15bea2064e1c414fb23182f6bc324bfd: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:16.857689 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:16.857736 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7ddae9ad2184cb9bd097d68171c3fc4: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:16.880805 9505 rebalancer_tool.cc:1084] tablet 1e44704a9b9c46cf954d8e6937983841: '31b02f06be8a4455b94b1254d77005bd' -> '3b69c5d799e14e438df760453438aa75' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:00:16.880937 9505 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:00:17.586419 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba8a985906c04117a440386d3011b936: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:17.614061 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22d40ca8e4714c2495974e28872c5687: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:17.614253 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2427aea1b9a6436ca7bd1796a85634ff: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:17.614375 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:17.614434 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ae84a8bf8ac745fb982412e9b194d7fc: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:17.614477 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba8a985906c04117a440386d3011b936: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:17.614521 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:17.614585 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22d40ca8e4714c2495974e28872c5687: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:17.614635 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:17.614692 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:17.614748 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7ddae9ad2184cb9bd097d68171c3fc4: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:17.614809 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22d40ca8e4714c2495974e28872c5687: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:17.614862 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:17.614980 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 61deeb642b0b4f06a2ef4b1a391fdcb9: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:17.615044 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15bea2064e1c414fb23182f6bc324bfd: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:17.615111 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d578050a40e0474484896c98e4da9f5a: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:17.615159 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ae84a8bf8ac745fb982412e9b194d7fc: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1c8cb9b1059a47cc822ca13a2b931776
W20240501 08:00:17.615298 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba8a985906c04117a440386d3011b936: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:17.615378 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 1807c0c2e7d04b20b0cda311c42eef8e
W20240501 08:00:17.615443 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7ddae9ad2184cb9bd097d68171c3fc4: could not find any suitable replica to move from server 09a7b6e2731a474b80411efb62323604 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:17.615504 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22d40ca8e4714c2495974e28872c5687: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server 1c8cb9b1059a47cc822ca13a2b931776
I20240501 08:00:17.889410 9505 rebalancer_tool.cc:1074] tablet cd07a00b6817400f86bf8c32bd592d8b: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '1807c0c2e7d04b20b0cda311c42eef8e' move scheduled
W20240501 08:00:18.128103 10911 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.0.99.129:40263, user_credentials={real_user=slave}} blocked reactor thread for 191841us
I20240501 08:00:18.192950 9505 rebalancer_tool.cc:1074] tablet 1088a116ccec4020aec562a69791a188: '31b02f06be8a4455b94b1254d77005bd' -> '3b69c5d799e14e438df760453438aa75' move scheduled
I20240501 08:00:18.299513 9505 rebalancer_tool.cc:1074] tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7: 'b4f8020266b5455b872a2973fc396578' -> 'd8aa5ac376a54e1f8417e5af11d28fa7' move scheduled
I20240501 08:00:18.403844 9505 rebalancer_tool.cc:1074] tablet 5d18a2918c75467a9fe59979ce8936cd: '09a7b6e2731a474b80411efb62323604' -> '1c8cb9b1059a47cc822ca13a2b931776' move scheduled
I20240501 08:00:18.670643 9505 rebalancer_tool.cc:1074] tablet e178f7c3c17c4e1ca3d16b6441f76e77: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '1c8cb9b1059a47cc822ca13a2b931776' move scheduled
I20240501 08:00:18.810053 9505 rebalancer_tool.cc:1084] tablet 81ccdd855b584d65860fd7a5777920a3: '31b02f06be8a4455b94b1254d77005bd' -> '1807c0c2e7d04b20b0cda311c42eef8e' move ignored: Illegal state: tablet 81ccdd855b584d65860fd7a5777920a3: replica 1807c0c2e7d04b20b0cda311c42eef8e is already present
I20240501 08:00:18.810130 9505 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:00:19.869243 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 3b69c5d799e14e438df760453438aa75
I20240501 08:00:19.930055 9505 rebalancer_tool.cc:1084] tablet 81ccdd855b584d65860fd7a5777920a3: 'd6b4be7bbd1649d0b7c5e8efe81f759f' -> '' move ignored: Not found: Peer with uuid d6b4be7bbd1649d0b7c5e8efe81f759f not found in consensus config
I20240501 08:00:19.930166 9505 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:00:20.497313 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 82bf711832394b1abda2824bb95c9358: could not find any suitable replica to move from server 31b02f06be8a4455b94b1254d77005bd to server 3b69c5d799e14e438df760453438aa75
W20240501 08:00:20.497468 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 013d2bd412784602ac81aa27c405ec0d: could not find any suitable replica to move from server b4f8020266b5455b872a2973fc396578 to server d8aa5ac376a54e1f8417e5af11d28fa7
W20240501 08:00:20.497509 9505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7ddae9ad2184cb9bd097d68171c3fc4: could not find any suitable replica to move from server d6b4be7bbd1649d0b7c5e8efe81f759f to server 1807c0c2e7d04b20b0cda311c42eef8e
I20240501 08:00:20.549743 9505 rebalancer_tool.cc:1074] tablet bde42e485c4b49ce8acd694a25ba683a: '31b02f06be8a4455b94b1254d77005bd' -> 'b4f8020266b5455b872a2973fc396578' move scheduled
I20240501 08:00:20.587917 9505 rebalancer_tool.cc:1084] tablet 8311d0921e9e417090e441884615525a: '09a7b6e2731a474b80411efb62323604' -> '1807c0c2e7d04b20b0cda311c42eef8e' move ignored: Illegal state: tablet 8311d0921e9e417090e441884615525a: replica 1807c0c2e7d04b20b0cda311c42eef8e is already present
I20240501 08:00:20.587987 9505 rebalancer_tool.cc:756] re-synchronizing cluster state
I20240501 08:00:21.392401 9505 rebalancer_tool.cc:1074] tablet 02b42b5fd3d440ea950b35ed461b8e12: 'd8aa5ac376a54e1f8417e5af11d28fa7' -> '1807c0c2e7d04b20b0cda311c42eef8e' move scheduled
I20240501 08:00:21.659446 9505 rebalancer_tool.cc:1074] tablet dd17760cd5c2454db97610b2f71c0ee2: '09a7b6e2731a474b80411efb62323604' -> '31b02f06be8a4455b94b1254d77005bd' move scheduled
I20240501 08:00:21.781649 9505 rebalancer_tool.cc:1084] tablet 5970772c02d04e0b95a2f2feb7dcd655: '3b69c5d799e14e438df760453438aa75' -> '1807c0c2e7d04b20b0cda311c42eef8e' move ignored: Not found: Peer with uuid 3b69c5d799e14e438df760453438aa75 not found in consensus config
I20240501 08:00:21.781741 9505 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:00:22.165010 12477 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:00:22.022918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:42587 (local address 127.0.0.1:58878)
0501 08:00:22.023096 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0501 08:00:22.023110 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 08:00:22.023190 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:00:22.164264 (+141074us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
Illegal state: tablet server 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): unacceptable health status UNAVAILABLE
I20240501 08:00:22.215260 12302 tablet_bootstrap.cc:492] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:22.215564 12302 tablet_bootstrap.cc:492] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap complete.
I20240501 08:00:22.215734 12302 ts_tablet_manager.cc:1367] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.042s user 0.024s sys 0.000s
I20240501 08:00:22.216146 12302 raft_consensus.cc:348] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } }
I20240501 08:00:22.216255 12302 raft_consensus.cc:729] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: LEARNER
I20240501 08:00:22.216352 12302 consensus_queue.cc:260] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } }
I20240501 08:00:22.216629 12302 ts_tablet_manager.cc:1398] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:22.217211 7467 tablet_copy_service.cc:342] P d6b4be7bbd1649d0b7c5e8efe81f759f: Request end of tablet copy session 31b02f06be8a4455b94b1254d77005bd-dd17760cd5c2454db97610b2f71c0ee2 received from {username='slave'} at 127.0.99.131:53791
I20240501 08:00:22.217298 7467 tablet_copy_service.cc:433] P d6b4be7bbd1649d0b7c5e8efe81f759f: ending tablet copy session 31b02f06be8a4455b94b1254d77005bd-dd17760cd5c2454db97610b2f71c0ee2 on tablet dd17760cd5c2454db97610b2f71c0ee2 with peer 31b02f06be8a4455b94b1254d77005bd
I20240501 08:00:22.218350 12500 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:22.218714 12500 tablet_bootstrap.cc:492] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: Bootstrap complete.
I20240501 08:00:22.218904 12500 ts_tablet_manager.cc:1367] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20240501 08:00:22.219352 12500 raft_consensus.cc:348] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
I20240501 08:00:22.219462 12500 raft_consensus.cc:729] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: LEARNER
I20240501 08:00:22.219537 12500 consensus_queue.cc:260] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
I20240501 08:00:22.219772 12500 ts_tablet_manager.cc:1398] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:22.220250 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session b4f8020266b5455b872a2973fc396578-4dde9710a5eb4baa881d9dd04affe184 received from {username='slave'} at 127.0.99.129:43719
I20240501 08:00:22.220340 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session b4f8020266b5455b872a2973fc396578-4dde9710a5eb4baa881d9dd04affe184 on tablet 4dde9710a5eb4baa881d9dd04affe184 with peer b4f8020266b5455b872a2973fc396578
I20240501 08:00:22.223393 10907 raft_consensus.cc:1050] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER b4f8020266b5455b872a2973fc396578 to VOTER
I20240501 08:00:22.223762 10907 consensus_queue.cc:237] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } }
I20240501 08:00:22.225075 9319 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 08:00:22.225080 7313 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 08:00:22.225497 11154 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:00:22.225687 11203 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240501 08:00:22.226024 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:22.227484 11203 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:00:22.227954 7313 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:00:22.230113 9319 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:00:22.230499 7171 catalog_manager.cc:5484] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 13 to 14, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
W20240501 08:00:22.231024 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:22.261157 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:22.283716 10907 raft_consensus.cc:1050] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 to VOTER
I20240501 08:00:22.284078 10907 consensus_queue.cc:237] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:22.285522 9319 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:22.286039 11880 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:22.286217 7707 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 08:00:22.286435 7513 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:22.286717 11880 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:22.288475 10907 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:22.288898 7707 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:22.290462 9319 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:00:22.291221 7169 catalog_manager.cc:5484] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 7 to 8, 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:00:22.320624 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:00:22.328605 7707 raft_consensus.cc:1203] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 08:00:22.362311 7575 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "28baea6d7d3f442ba1dbfc5b37de90ac"
dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52664
I20240501 08:00:22.362516 7575 raft_consensus.cc:595] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:22.394425 12302 ts_tablet_manager.cc:902] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:00:22.394986 12302 tablet_copy_client.cc:308] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:00:22.395531 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626 from peer 31b02f06be8a4455b94b1254d77005bd ({username='slave'} at 127.0.99.131:52783)
I20240501 08:00:22.395625 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626 from peer 31b02f06be8a4455b94b1254d77005bd at {username='slave'} at 127.0.99.131:52783: session id = 31b02f06be8a4455b94b1254d77005bd-b8fdc2b9c2ef44b0bfec59a1a2f6b626
I20240501 08:00:22.396366 7336 tablet_copy_source_session.cc:205] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:22.397162 12302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:22.398988 12302 tablet_copy_client.cc:783] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:22.399143 12302 tablet_copy_client.cc:647] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:22.400312 12302 tablet_copy_client.cc:523] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:22.401589 12302 tablet_bootstrap.cc:492] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap starting.
W20240501 08:00:22.405848 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:22.421370 12302 tablet_bootstrap.cc:492] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:22.421659 12302 tablet_bootstrap.cc:492] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd: Bootstrap complete.
I20240501 08:00:22.421834 12302 ts_tablet_manager.cc:1367] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.003s
I20240501 08:00:22.422298 12302 raft_consensus.cc:348] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } }
I20240501 08:00:22.422416 12302 raft_consensus.cc:729] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Initialized, Role: LEARNER
I20240501 08:00:22.422497 12302 consensus_queue.cc:260] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: true } }
I20240501 08:00:22.422750 12302 ts_tablet_manager.cc:1398] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 08:00:22.423323 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session 31b02f06be8a4455b94b1254d77005bd-b8fdc2b9c2ef44b0bfec59a1a2f6b626 received from {username='slave'} at 127.0.99.131:52783
I20240501 08:00:22.423429 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session 31b02f06be8a4455b94b1254d77005bd-b8fdc2b9c2ef44b0bfec59a1a2f6b626 on tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626 with peer 31b02f06be8a4455b94b1254d77005bd
I20240501 08:00:22.562358 12552 raft_consensus.cc:1050] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER 31b02f06be8a4455b94b1254d77005bd to VOTER
I20240501 08:00:22.562808 12552 consensus_queue.cc:237] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } }
I20240501 08:00:22.564245 7575 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:22.564263 9318 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:22.564252 9056 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:22.564734 10306 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:22.564939 12555 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:22.565302 10306 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:22.567279 12555 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:00:22.567696 7575 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:00:22.568156 9056 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:00:22.571161 7171 catalog_manager.cc:5484] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 10 to 11, 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:00:22.571969 9318 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:00:22.574542 7313 consensus_queue.cc:237] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } }
I20240501 08:00:22.575891 7575 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:00:22.575909 9056 raft_consensus.cc:1261] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:00:22.576498 12554 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:00:22.577476 12552 consensus_queue.cc:1035] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:00:22.579466 12552 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:00:22.580571 9056 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:00:22.581041 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e178f7c3c17c4e1ca3d16b6441f76e77 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:22.582820 7167 catalog_manager.cc:5484] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 11 to 12, VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:22.586334 9298 tablet_service.cc:1514] Processing DeleteTablet for tablet e178f7c3c17c4e1ca3d16b6441f76e77 with delete_type TABLET_DATA_TOMBSTONED (TS 1c8cb9b1059a47cc822ca13a2b931776 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:40266
I20240501 08:00:22.586470 7575 raft_consensus.cc:2932] T e178f7c3c17c4e1ca3d16b6441f76e77 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:00:22.586727 12570 tablet_replica.cc:330] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776: stopping tablet replica
I20240501 08:00:22.586891 12570 raft_consensus.cc:2229] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:22.587060 12570 raft_consensus.cc:2258] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:22.587936 12570 ts_tablet_manager.cc:1875] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:22.589952 12570 ts_tablet_manager.cc:1888] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240501 08:00:22.590090 12570 log.cc:1196] T e178f7c3c17c4e1ca3d16b6441f76e77 P 1c8cb9b1059a47cc822ca13a2b931776: Deleting WAL directory at /tmp/dist-test-taskS8gUPX/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714550363394914-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/e178f7c3c17c4e1ca3d16b6441f76e77
I20240501 08:00:22.590626 7159 catalog_manager.cc:4830] TS 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): tablet e178f7c3c17c4e1ca3d16b6441f76e77 (table rebalance_test_table_5 [id=2427aea1b9a6436ca7bd1796a85634ff]) successfully deleted
W20240501 08:00:22.594099 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:22.610615 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:00:22.617209 7575 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "28baea6d7d3f442ba1dbfc5b37de90ac"
dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52692
I20240501 08:00:22.617347 7575 raft_consensus.cc:595] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:22.679721 7313 raft_consensus.cc:1203] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20240501 08:00:22.724315 10907 raft_consensus.cc:979] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd: : Instructing follower 09a7b6e2731a474b80411efb62323604 to start an election
I20240501 08:00:22.724452 10907 raft_consensus.cc:1067] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Signalling peer 09a7b6e2731a474b80411efb62323604 to start an election
I20240501 08:00:22.724952 7707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "28baea6d7d3f442ba1dbfc5b37de90ac"
dest_uuid: "09a7b6e2731a474b80411efb62323604"
from {username='slave'} at 127.0.99.131:57527
I20240501 08:00:22.725090 7707 raft_consensus.cc:482] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:22.725152 7707 raft_consensus.cc:3037] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:22.726080 7707 raft_consensus.cc:504] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:22.726518 7707 leader_election.cc:288] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 2 election: Requested vote from peers 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861), 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589), 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471)
I20240501 08:00:22.726925 9318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28baea6d7d3f442ba1dbfc5b37de90ac" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1c8cb9b1059a47cc822ca13a2b931776"
I20240501 08:00:22.727084 7575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28baea6d7d3f442ba1dbfc5b37de90ac" candidate_uuid: "09a7b6e2731a474b80411efb62323604" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
I20240501 08:00:22.727120 9318 raft_consensus.cc:3037] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:22.727207 7575 raft_consensus.cc:3032] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:22.727273 7575 raft_consensus.cc:729] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Running, Role: LEADER
I20240501 08:00:22.727448 7575 consensus_queue.cc:260] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
W20240501 08:00:22.727525 7644 leader_election.cc:334] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:00:22.727641 7575 raft_consensus.cc:3037] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:22.728291 9318 raft_consensus.cc:2454] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 2.
I20240501 08:00:22.728624 7575 raft_consensus.cc:2454] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a7b6e2731a474b80411efb62323604 in term 2.
I20240501 08:00:22.729128 7643 leader_election.cc:302] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, 1c8cb9b1059a47cc822ca13a2b931776, 31b02f06be8a4455b94b1254d77005bd; no voters: 3b69c5d799e14e438df760453438aa75
I20240501 08:00:22.729290 11088 raft_consensus.cc:2781] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:22.729386 11088 raft_consensus.cc:686] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 2 LEADER]: Becoming Leader. State: Replica: 09a7b6e2731a474b80411efb62323604, State: Running, Role: LEADER
I20240501 08:00:22.729527 11088 consensus_queue.cc:237] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:00:22.731705 7167 catalog_manager.cc:5484] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 reported cstate change: term changed from 1 to 2, leader changed from 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) to 09a7b6e2731a474b80411efb62323604 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 08:00:22.761593 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:00:22.761952 7575 raft_consensus.cc:1203] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:00:22.783804 7575 raft_consensus.cc:1203] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:00:22.901430 10907 raft_consensus.cc:1050] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER 09a7b6e2731a474b80411efb62323604 to VOTER
I20240501 08:00:22.901840 10907 consensus_queue.cc:237] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } }
I20240501 08:00:22.903224 9056 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:22.903278 7707 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:00:22.903689 11154 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:22.903833 12627 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 08:00:22.903975 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:22.905649 12627 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 09a7b6e2731a474b80411efb62323604 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
I20240501 08:00:22.906087 7707 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 09a7b6e2731a474b80411efb62323604 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
I20240501 08:00:22.907975 9056 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 09a7b6e2731a474b80411efb62323604 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
I20240501 08:00:22.908541 7169 catalog_manager.cc:5484] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 10 to 11, 09a7b6e2731a474b80411efb62323604 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:00:22.921330 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5970772c02d04e0b95a2f2feb7dcd655: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:23.094390 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:00:23.125802 11088 raft_consensus.cc:1050] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER b4f8020266b5455b872a2973fc396578 to VOTER
I20240501 08:00:23.126269 11088 consensus_queue.cc:237] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } }
I20240501 08:00:23.127712 9056 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 08:00:23.127707 7313 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 08:00:23.128091 11088 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:00:23.128257 11072 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240501 08:00:23.128398 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:23.130221 11088 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:00:23.131479 7313 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:00:23.133318 9056 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:00:23.133956 7167 catalog_manager.cc:5484] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 13 to 14, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:00:23.194931 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:23.234081 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:00:23.271862 12666 raft_consensus.cc:1050] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f: attempting to promote NON_VOTER 31b02f06be8a4455b94b1254d77005bd to VOTER
I20240501 08:00:23.272274 12666 consensus_queue.cc:237] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } }
I20240501 08:00:23.273546 7575 raft_consensus.cc:1261] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:23.273679 7707 raft_consensus.cc:1261] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:23.273943 12669 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:23.274156 12666 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:00:23.274794 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:23.275655 9318 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P 1c8cb9b1059a47cc822ca13a2b931776 [term 2 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:00:23.276048 11088 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:23.276289 12666 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:00:23.276839 7707 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:00:23.279153 7169 catalog_manager.cc:5484] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 7 to 8, 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:23.279884 7575 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
W20240501 08:00:23.293061 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:23.310750 12554 raft_consensus.cc:1050] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER 31b02f06be8a4455b94b1254d77005bd to VOTER
I20240501 08:00:23.311251 12554 consensus_queue.cc:237] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } }
I20240501 08:00:23.312932 7707 raft_consensus.cc:1261] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:23.313755 7575 raft_consensus.cc:1261] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:23.314026 7577 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 2 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:00:23.314425 11088 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:23.314721 12694 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:00:23.315086 7251 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:23.315325 12695 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:23.318362 12694 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:00:23.320634 7707 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:00:23.322348 7575 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:00:23.322926 7169 catalog_manager.cc:5484] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 7 to 8, 31b02f06be8a4455b94b1254d77005bd (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:23.343122 7313 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8fdc2b9c2ef44b0bfec59a1a2f6b626"
dest_uuid: "b4f8020266b5455b872a2973fc396578"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52532
I20240501 08:00:23.343298 7313 raft_consensus.cc:595] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Received request to transfer leadership
W20240501 08:00:23.445901 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:23.567080 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:00:23.579350 7313 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8fdc2b9c2ef44b0bfec59a1a2f6b626"
dest_uuid: "b4f8020266b5455b872a2973fc396578"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52550
I20240501 08:00:23.579499 7313 raft_consensus.cc:595] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Received request to transfer leadership
W20240501 08:00:23.635205 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:23.662752 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:23.811390 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:23.816046 7313 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8fdc2b9c2ef44b0bfec59a1a2f6b626"
dest_uuid: "b4f8020266b5455b872a2973fc396578"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52572
I20240501 08:00:23.816185 7313 raft_consensus.cc:595] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Received request to transfer leadership
W20240501 08:00:23.840744 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:00:23.881682 12695 raft_consensus.cc:979] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578: : Instructing follower 31b02f06be8a4455b94b1254d77005bd to start an election
I20240501 08:00:23.881800 12695 raft_consensus.cc:1067] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Signalling peer 31b02f06be8a4455b94b1254d77005bd to start an election
I20240501 08:00:23.882376 7575 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8fdc2b9c2ef44b0bfec59a1a2f6b626"
dest_uuid: "31b02f06be8a4455b94b1254d77005bd"
from {username='slave'} at 127.0.99.129:41265
I20240501 08:00:23.882557 7575 raft_consensus.cc:482] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:23.882627 7575 raft_consensus.cc:3037] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:23.883692 7575 raft_consensus.cc:504] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } }
I20240501 08:00:23.884179 7575 leader_election.cc:288] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 2 election: Requested vote from peers 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789), b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263), 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589)
I20240501 08:00:23.884655 7707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8fdc2b9c2ef44b0bfec59a1a2f6b626" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "09a7b6e2731a474b80411efb62323604"
I20240501 08:00:23.884709 7313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8fdc2b9c2ef44b0bfec59a1a2f6b626" candidate_uuid: "31b02f06be8a4455b94b1254d77005bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b4f8020266b5455b872a2973fc396578"
I20240501 08:00:23.884795 7707 raft_consensus.cc:3037] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:23.884835 7313 raft_consensus.cc:3032] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:23.884888 7313 raft_consensus.cc:729] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Running, Role: LEADER
I20240501 08:00:23.885054 7313 consensus_queue.cc:260] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } }
I20240501 08:00:23.885226 7313 raft_consensus.cc:3037] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:23.885700 7707 raft_consensus.cc:2454] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 2.
I20240501 08:00:23.886087 7313 raft_consensus.cc:2454] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b02f06be8a4455b94b1254d77005bd in term 2.
I20240501 08:00:23.886437 7515 leader_election.cc:302] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 09a7b6e2731a474b80411efb62323604, 31b02f06be8a4455b94b1254d77005bd, b4f8020266b5455b872a2973fc396578; no voters:
I20240501 08:00:23.886615 12772 raft_consensus.cc:2781] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [term 2 FOLLOWER]: Leader election won for term 2
W20240501 08:00:23.886619 7513 leader_election.cc:334] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:00:23.886703 12772 raft_consensus.cc:686] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [term 2 LEADER]: Becoming Leader. State: Replica: 31b02f06be8a4455b94b1254d77005bd, State: Running, Role: LEADER
I20240501 08:00:23.886873 12772 consensus_queue.cc:237] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } }
I20240501 08:00:23.888573 7171 catalog_manager.cc:5484] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: term changed from 1 to 2, leader changed from b4f8020266b5455b872a2973fc396578 (127.0.99.129) to 31b02f06be8a4455b94b1254d77005bd (127.0.99.131). New cstate: current_term: 2 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:00:23.892118 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:23.996006 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5970772c02d04e0b95a2f2feb7dcd655: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:24.144939 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:24.170135 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:24.178722 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:24.187007 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:24.240787 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:24.249253 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:00:24.278791 7313 raft_consensus.cc:1261] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 2 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:00:24.279250 12772 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:00:24.315387 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:24.319720 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:24.357174 7707 raft_consensus.cc:1261] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 2 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:00:24.357625 12772 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:00:24.362186 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:24.416198 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:24.426760 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:24.431149 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:24.525790 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:24.608523 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:24.681190 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:24.708863 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:24.720494 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:24.819260 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:24.825323 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:25.271751 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:25.294284 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:25.307430 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:25.451047 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:25.696851 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:25.698194 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:25.792708 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:25.832648 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:25.915019 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:25.959326 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:26.004820 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:26.067373 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5970772c02d04e0b95a2f2feb7dcd655: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:26.087671 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:26.223968 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:26.273505 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:26.284699 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:26.302915 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:26.563277 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:26.595005 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:26.645512 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:26.666566 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:26.707160 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:00:26.758623 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:00:26.780191 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240501 08:00:26.780877 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:26.803253 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:26.808705 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:26.819509 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:26.827435 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:00:26.828022 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:26.851596 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:26.870965 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:26.999711 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:27.121260 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:27.134519 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:27.200340 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:27.260489 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:27.275874 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:27.339489 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:27.397918 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:27.418224 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:27.711230 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:27.748629 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:27.872351 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:28.153749 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:28.235242 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:28.263753 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:28.337148 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:28.360881 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:28.366969 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:28.421626 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:28.640662 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:28.747761 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:28.785501 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:28.845571 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:28.849670 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:28.963968 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:29.005923 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:29.055179 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:29.154947 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:29.174379 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:29.181718 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:29.211216 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:29.322456 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:29.354703 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:29.389312 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:29.402976 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:29.549628 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:29.551453 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:29.667163 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:29.744065 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:29.763660 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:29.808156 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:29.840975 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:29.866851 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:29.996584 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:30.084298 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:30.177367 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:30.177431 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 08:00:30.178153 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5970772c02d04e0b95a2f2feb7dcd655: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:30.294150 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:30.309335 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:30.316335 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:30.579300 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:30.700538 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:30.735582 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:30.736497 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:30.761780 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:31.001785 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:31.124497 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:31.180372 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:31.199728 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:31.319003 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:31.458423 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:31.535280 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:31.589787 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:31.636839 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:31.693048 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:31.717067 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:31.752633 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:00:31.753300 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:31.762841 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:00:31.763571 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:31.874533 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:00:31.875459 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:31.925451 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:31.931830 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:00:31.962216 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:31.968533 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:31.973806 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:32.010746 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:32.104708 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:32.189908 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:32.200857 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:32.255344 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:32.255702 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:32.293180 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:32.574544 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:32.637382 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:32.689705 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:32.729840 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:32.768069 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:32.972352 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:33.018565 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:33.088194 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:33.130021 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:33.172382 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:33.248989 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:33.291467 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:33.410275 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:33.555600 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:33.578787 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:33.602319 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:33.886911 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:33.928014 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:33.970867 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:34.063629 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:34.105506 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:34.114522 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:34.133244 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:34.170339 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:34.185405 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:34.194429 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:34.410339 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:34.475919 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:34.497356 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:34.572547 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:34.615563 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:34.689702 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:34.731264 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:34.752557 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:34.767402 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:34.770064 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:35.134728 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:35.170846 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:35.239049 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:35.278374 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:35.396343 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:35.512499 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:35.609484 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:35.628372 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:35.654862 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:35.746418 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:35.760569 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:35.793449 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:35.866394 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:36.030849 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:36.136444 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:36.164585 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:36.434558 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:36.475332 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:36.486747 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:36.581830 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:36.613101 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:36.618777 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:36.628983 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:36.710742 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:36.731171 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:36.771245 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:00:36.771903 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:36.812752 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:36.846668 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:00:36.933353 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:36.944769 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:00:36.945515 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:37.047992 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:37.081904 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:00:37.096696 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:37.102392 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:37.173758 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:37.193666 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:37.273527 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:37.345827 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:37.424459 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:37.512398 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:37.651829 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:37.697064 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:37.783350 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:37.921175 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:37.939605 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:37.996570 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:38.005856 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:38.036720 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:38.082530 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:38.224478 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:38.293952 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:38.398779 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:38.399509 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5970772c02d04e0b95a2f2feb7dcd655: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:38.529445 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:38.616459 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:38.647416 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:38.864315 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:38.932878 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:38.999421 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:39.027742 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:39.039011 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:39.048391 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:39.067502 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:39.175432 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:39.310979 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:39.409276 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:39.419677 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:39.503122 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:39.541791 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:39.569690 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:39.601132 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:39.605465 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:39.647488 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:39.690237 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:39.793437 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:39.914862 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:39.949409 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:40.079974 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:40.175597 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:40.260113 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:40.293293 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:40.365033 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:40.393507 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:40.659952 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:40.693411 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:40.744910 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:40.754091 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:40.801033 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:40.863523 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:40.958086 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:41.072419 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:41.143908 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:41.148947 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:41.263025 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:41.286885 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:41.497740 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:41.525902 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:41.537506 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:41.659992 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:41.665282 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:41.682883 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:41.720316 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:41.766036 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:41.808662 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:00:41.809320 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:41.911721 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:00:41.991636 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 08:00:41.992282 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:41.997287 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:42.032753 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:42.064411 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:42.115363 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 08:00:42.115971 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:42.134531 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:42.185592 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:42.311458 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:42.356783 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:42.467787 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:42.511204 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:42.661967 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:42.723620 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:42.746951 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:42.831744 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:42.957513 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:43.035394 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:43.167455 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:43.336831 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:43.338017 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:43.359990 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:43.564265 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:43.587340 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:43.657620 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:43.677968 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:43.690857 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:43.738871 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:43.804304 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:43.853927 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:44.075035 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:44.170014 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:44.254118 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:44.265712 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:44.295315 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:44.315877 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:44.341009 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:44.370589 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:44.455013 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:44.506202 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:44.578465 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:44.677299 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:44.678808 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:44.827703 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:44.870628 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:44.878875 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:44.932857 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:44.997313 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:45.129348 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:45.231446 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:45.284663 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:45.304296 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:45.390637 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:45.458222 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:45.512459 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:45.744709 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:45.888654 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:46.050343 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:46.067574 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:46.107666 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:46.111675 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:46.136785 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:46.242694 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:46.268159 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:46.271168 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:46.357474 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:46.569491 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:46.651044 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:46.733834 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:46.741701 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:46.742801 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:46.753499 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:46.790510 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:46.824700 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:00:46.825361 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:46.858170 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:46.909865 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:47.028894 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:00:47.050209 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:00:47.152280 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 08:00:47.152966 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:47.258688 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:47.294870 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:47.306517 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:47.318847 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:47.452464 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:47.505100 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:47.729370 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:47.732234 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:47.808066 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:47.869895 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:47.911837 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:47.915614 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:48.166564 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:48.292335 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:48.305586 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:48.356242 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:48.504271 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:48.527477 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:48.591987 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:48.648365 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:48.711073 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:48.774091 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:48.844878 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:48.893738 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:48.923156 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:48.928637 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:49.001749 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:49.086230 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:49.122818 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:49.130228 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:49.368110 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:49.376425 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:49.379920 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:49.438138 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:49.604851 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:49.650895 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:49.708215 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:49.756742 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:49.806118 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:49.842522 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:49.937043 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:49.973477 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:50.065070 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:50.069339 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:50.209671 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:50.300657 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:50.314272 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:50.422040 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:50.689263 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:50.722134 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:50.798061 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:50.803349 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:50.817937 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:50.937314 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:51.106451 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:51.124570 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:51.212033 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:51.213673 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:51.311726 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:51.321697 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:51.411758 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:51.422967 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:51.475932 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:51.624487 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:51.643662 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:51.801678 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:51.860479 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:51.873871 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:00:51.884593 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:52.006649 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:52.030269 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:52.040887 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:52.046661 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:52.076936 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:00:52.077708 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:52.085026 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:00:52.210894 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:52.224135 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:00:52.224949 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:52.342608 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:52.425714 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:52.468225 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:52.473407 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:52.611641 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:52.775208 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:52.867374 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:52.964756 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:53.057698 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:53.155253 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:53.184979 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:53.264775 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:53.265487 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:53.490135 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:53.599242 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:53.641852 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:53.644663 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:53.646229 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:53.693645 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:53.723693 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:53.808645 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:53.947757 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:54.014137 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:54.198169 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:54.256592 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:54.345369 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:54.387806 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:54.515893 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:54.529791 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:54.534866 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:54.596709 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:54.617354 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:54.678893 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:54.712321 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:54.824421 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:54.825160 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5970772c02d04e0b95a2f2feb7dcd655: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:54.843214 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:54.873689 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:54.898985 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:54.947227 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:55.161217 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:55.262466 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:55.262722 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:55.294718 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:55.391754 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:55.509472 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:55.694080 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:55.726841 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:55.730856 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:55.754736 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:56.007678 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:56.102197 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:56.154196 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:56.157099 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:56.173707 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:56.202867 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:56.238731 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:56.255992 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:56.295454 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:56.482079 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:56.593931 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:56.655717 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:56.699512 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:56.819242 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:56.835584 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:56.864645 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:56.876219 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:00:56.953255 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:57.114538 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240501 08:00:57.123425 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:57.149291 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 36 similar messages]
W20240501 08:00:57.204893 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:57.215783 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:57.224238 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:00:57.226692 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:57.283900 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:57.334990 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:57.358516 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:57.398767 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:57.488791 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:57.493022 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:57.556880 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:57.749449 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:57.865789 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:58.010172 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:58.021795 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:58.054759 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:58.147246 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:58.338045 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:58.341331 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:58.477846 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:58.504040 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:58.522722 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:58.542539 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:58.585197 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:58.693537 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:58.773070 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:58.818579 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:58.967532 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:59.019012 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:59.049702 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:59.091102 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:59.100960 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:59.148231 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:59.329872 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:59.366972 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:59.486394 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:59.570513 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:59.575145 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:00:59.593808 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:59.644306 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:00:59.669431 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:59.798696 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:59.840996 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:00:59.920739 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:59.949414 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:59.967685 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:59.972195 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:00.177906 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:00.224368 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:00.283546 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:01:00.550631 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:00.616199 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:00.643348 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:00.748952 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:01:00.749933 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:01:00.841353 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:01:00.993865 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:01.007584 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:01.016357 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:01:01.116894 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:01.126266 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:01:01.226979 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:01.363559 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:01.374104 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:01.424899 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:01.487700 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:01.513939 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:01.647311 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:01.679843 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:01.725934 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:01.804994 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:01:01.891713 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:01.894428 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:01:01.996610 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:02.031940 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:02.089749 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:02.149394 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:02.188601 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:01:02.189304 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:02.192097 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:01:02.193186 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:02.261337 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:02.334062 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:01:02.436959 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:02.441599 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:02.495759 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:02.523799 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:02.651430 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:02.739295 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:02.751297 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:02.997568 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:03.042093 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:03.049752 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:03.325620 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:03.336302 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:03.397094 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:03.424207 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:03.494827 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:03.559046 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:03.612200 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:03.749035 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:03.804437 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:03.819665 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:03.946653 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:03.970978 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:04.059516 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:04.070122 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:04.187407 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:04.346421 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:01:04.416082 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:04.467707 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:04.487782 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:04.491065 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:04.557193 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:04.674892 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:04.751437 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:04.794449 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:04.847159 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:04.863516 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:04.946187 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:04.986979 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:05.071494 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:05.190255 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:05.206085 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:05.367480 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:05.374024 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:05.380137 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:05.499955 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:05.706503 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:05.720429 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:05.790846 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:05.848304 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:05.948271 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:06.017621 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:06.088456 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:06.103717 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:06.124946 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:06.128798 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:06.296026 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:06.461135 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:06.479379 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:06.502377 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:06.538842 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:06.771175 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:06.818035 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:01:06.873309 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:06.922672 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:01:06.969679 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:07.003763 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:07.012476 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:07.108688 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:07.148783 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:07.193348 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:01:07.228044 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:07.269177 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:07.361773 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:01:07.362497 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:07.386670 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:01:07.407209 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:07.415093 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:07.529863 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:07.539382 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:07.574426 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:07.813560 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:07.832782 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:07.901103 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:07.918442 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:07.994661 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:08.052577 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:08.175940 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:08.283461 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:08.334403 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:08.383906 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:08.509136 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:08.512974 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:08.523427 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:08.532049 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:08.692272 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:08.899757 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:08.905948 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:08.978421 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:09.072307 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:09.196354 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:09.250180 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:01:09.366779 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:09.385200 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:09.483352 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:09.509452 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:09.510167 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:09.510166 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:09.605525 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:09.622193 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:09.765013 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:09.765193 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:09.864537 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:09.997170 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:10.049489 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:10.098204 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:10.116966 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:10.187840 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:10.394302 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:10.553097 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:10.627015 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:10.645155 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:10.647138 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:10.665809 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:10.804646 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:10.826367 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:10.918903 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:10.972450 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:10.983573 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:11.141053 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:11.189711 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:11.263206 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:11.272413 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:11.437474 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:11.708880 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:11.737535 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:11.756810 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:11.820643 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:11.856730 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:11.859751 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:11.888756 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:01:11.912572 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:11.975317 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:12.032119 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:12.061930 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:01:12.103555 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:12.127039 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:12.210803 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:12.237520 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:12.351018 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:01:12.461424 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:01:12.462253 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:12.512343 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:12.622768 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:01:12.623482 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:12.661895 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:12.671505 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:12.743656 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:12.983131 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:13.110986 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:13.130316 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:13.217567 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:13.248570 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:13.296422 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:13.319758 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:13.485548 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:13.586161 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:13.599773 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:13.602397 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:13.651934 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:13.730324 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:13.783244 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:14.121867 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:14.199294 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:14.268678 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:14.278005 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:14.283957 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:14.343223 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:14.357060 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:14.375092 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:14.410542 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:14.437433 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:14.587908 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:14.615574 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:14.664656 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:14.731750 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:14.748092 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:14.914685 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:15.045547 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:15.051462 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:15.227490 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:15.293537 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:15.343365 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:15.346715 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:15.413627 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:15.489380 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:15.654454 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:15.742925 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:15.745960 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:15.864392 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:15.965155 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:16.044740 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:16.083051 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:16.133045 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:16.154605 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:16.186617 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:16.208779 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:16.451736 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:16.528553 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:16.568225 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:16.640920 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:16.714736 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:16.731591 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:16.736546 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:16.931890 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:16.957969 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:16.984249 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:17.063035 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:17.075806 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:17.101500 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:01:17.117975 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:17.156692 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:17.300737 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:17.353077 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:17.485543 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 08:01:17.486871 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:17.487004 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:01:17.573441 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:17.614466 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:17.644560 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:01:17.645241 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:17.677307 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:17.736550 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:17.994189 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:18.052265 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:18.067736 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:18.070653 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:18.261222 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:18.346828 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:18.391800 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:18.406733 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:18.489712 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:18.570045 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:18.612622 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:18.671902 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:18.705477 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:18.777170 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:19.011231 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:19.064954 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:19.118979 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:19.169751 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:19.228106 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:19.291085 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:19.393922 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:19.534777 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:19.577905 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:19.657464 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:19.698446 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:19.698452 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:19.734031 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:19.740667 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:19.778194 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:19.952845 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:20.015877 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:20.071167 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:20.232321 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:20.254983 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:20.292299 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:20.299243 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:20.472770 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:20.591194 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:20.762164 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:20.868263 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:20.967341 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:20.969383 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:20.975792 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:21.015714 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:21.123965 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:21.159802 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:21.162712 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:21.239421 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:21.384585 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:21.409066 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:21.482059 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:21.540452 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:21.688666 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:21.693718 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:21.866215 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:21.942848 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:21.974308 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:21.977281 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:21.985116 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:22.058707 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:22.074508 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:22.120837 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:01:22.243098 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:22.274940 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:22.383105 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:22.507833 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 08:01:22.517953 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:22.518508 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:01:22.572417 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:22.582866 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:22.768415 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 08:01:22.769093 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:22.843081 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:22.875070 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:22.896849 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:22.934459 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:22.990969 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:22.995468 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:23.148769 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:23.339766 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:23.417098 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:23.451189 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:23.516217 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:23.593426 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:23.599145 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:23.613097 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:23.674921 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:23.809605 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:23.917213 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:23.960602 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:23.989526 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:23.994822 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:24.008921 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:24.130174 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:24.265882 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:24.390437 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:24.444272 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:24.544886 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:24.575217 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:24.644857 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:24.658027 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:24.720593 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:24.771239 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:24.801306 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:24.857753 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:24.961824 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:25.165289 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:25.187508 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:25.262347 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:25.371872 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:25.374068 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:25.396473 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:25.520771 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:25.655045 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:25.734679 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:25.794967 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:25.953076 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:26.039847 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:26.052100 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:26.130527 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:26.146781 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:26.235499 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:26.249238 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:26.345170 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:26.403877 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:26.531497 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:26.557864 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:26.558934 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:26.740496 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:26.782826 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:26.815093 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:26.858181 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:26.915426 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:27.018357 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:27.071395 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:27.113785 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:27.132722 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:27.142015 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240501 08:01:27.236846 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:27.267560 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:27.273380 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:27.426532 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:27.504539 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:27.546250 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:01:27.598560 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:01:27.599207 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:27.603163 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:27.603981 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5970772c02d04e0b95a2f2feb7dcd655: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:27.637702 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:27.704706 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:27.781939 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:01:27.850425 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:27.934643 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:28.056982 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:28.112443 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:28.242569 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:28.367416 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:28.423765 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:28.478741 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:28.532881 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:28.559540 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:28.560324 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:28.642863 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:28.764489 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:28.824723 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:28.825191 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:28.979624 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:29.018250 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:29.029016 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:29.044250 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:29.135304 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:29.201581 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:29.214558 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:29.254951 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:29.413668 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:29.464476 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:29.609946 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:29.703275 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:29.823745 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:29.839828 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:29.861215 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:29.888191 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:29.901712 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:29.946308 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:30.083451 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:30.309876 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:30.427450 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:30.429085 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:30.446416 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:30.492985 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:30.587095 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:30.871587 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:30.923983 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:30.924186 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:30.946549 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:30.983161 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:31.000487 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:31.109017 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:31.229843 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:31.293452 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:31.302769 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:31.451252 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:31.473557 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:31.534704 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:31.538352 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:31.571882 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:31.738060 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:31.779202 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:31.840747 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:31.892783 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:32.066422 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:32.084106 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:32.096621 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:32.196225 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:01:32.258764 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:32.293146 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:32.311866 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:32.420922 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:32.466157 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:32.517252 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:32.532311 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:32.652666 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:01:32.653097 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:01:32.673856 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:32.696295 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:32.778156 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:32.810895 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:32.924995 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:01:33.080436 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:33.087829 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:33.283942 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:33.292915 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:33.359462 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:33.574882 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:33.597548 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:33.605609 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:33.634397 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:33.650619 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:33.650730 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:33.748163 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:33.748793 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:33.760584 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:33.840407 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:34.102877 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:34.201958 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:34.220407 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:34.220805 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:34.285068 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:34.324959 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:34.511847 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:34.650518 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:34.657359 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:34.683223 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:34.775527 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:34.822808 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:34.854285 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:34.858072 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:35.075405 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:35.106627 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:35.121914 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:35.174537 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:35.249595 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:35.293892 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:35.487388 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:35.555243 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:35.661257 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:35.774767 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:35.778149 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:35.859488 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:35.868548 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:35.973044 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:35.986912 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:36.006045 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:36.039660 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:36.079603 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:36.190742 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:36.195441 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:36.268422 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:36.365362 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:36.435189 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:36.596884 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:36.653329 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:36.767556 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:36.771915 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:36.877640 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:36.901007 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:36.960211 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:37.040483 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:37.140765 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:37.209031 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:37.285081 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:37.292196 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:01:37.347074 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:37.364748 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:37.386502 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:37.392356 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:37.570439 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:37.674865 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:37.679399 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:01:37.741573 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:01:37.742348 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:37.746578 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:37.746809 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:38.016640 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:01:38.074738 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:38.119626 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:38.166770 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:38.194494 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:38.321280 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:38.338162 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:38.353852 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:38.452322 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:38.550683 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:38.577534 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:38.590898 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:38.615561 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:38.690637 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:38.765045 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:38.830597 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:38.834889 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:38.973438 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:39.154798 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:39.193770 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:39.235800 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:39.331259 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:39.396474 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:39.445614 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:39.465214 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:39.654218 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:39.730746 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:39.785890 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:39.794950 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:39.883563 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:39.948171 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:39.973219 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:40.047739 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:40.154431 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:40.201927 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:40.232403 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:40.276799 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:40.329526 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:40.599287 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:40.679232 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:40.860553 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:40.889351 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:40.988552 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:40.991204 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:41.034381 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:41.051183 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:41.054682 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:41.085156 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:41.088867 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:41.109973 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:41.307646 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:41.382377 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:41.464430 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:41.486783 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:41.510500 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:41.601611 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:41.750532 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:41.887583 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:41.892731 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:41.931901 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:42.023140 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:42.138446 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:42.271386 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:42.289319 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:42.377250 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:42.378839 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:01:42.580822 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:42.590584 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:42.616050 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:42.618149 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:42.634208 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:42.637820 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:42.700459 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:42.817385 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:01:42.818403 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:42.853116 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:42.926151 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:01:43.078523 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:01:43.079402 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:43.104097 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:43.135862 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:43.199141 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:43.458972 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:43.470477 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:43.478173 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:43.565932 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:43.567323 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:43.615737 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:43.667450 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:43.677022 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:43.685154 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:43.728119 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:43.825711 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:43.841352 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:44.112833 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:44.182093 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:44.202365 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:44.224390 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:44.492189 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:44.562954 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:44.593787 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:44.645200 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:44.748137 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:44.790351 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:44.803210 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:44.863930 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:45.031734 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:45.037477 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:45.068125 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:45.143379 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:45.210798 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:45.227569 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:45.290406 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:45.346428 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:45.522405 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:45.524286 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:45.756469 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:45.777009 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:45.780731 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:45.846029 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:45.908180 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:45.964228 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:45.970463 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:46.007067 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:46.096910 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:46.227671 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:46.343941 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:46.349799 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:46.435424 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:46.496198 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:46.517163 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:46.548753 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:46.733661 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:46.885334 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:46.911058 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:47.054987 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:47.091553 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:47.132179 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:47.253633 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:47.269665 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:47.287025 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:47.379571 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:47.386210 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:01:47.386899 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:47.550611 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:47.623714 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:47.800354 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:47.811365 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:47.814306 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:47.814662 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:47.823233 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:01:47.833932 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:47.962421 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:48.102051 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:48.111840 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:01:48.198637 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:01:48.274351 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:48.277609 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:48.321472 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:48.375205 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:48.389686 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:48.488389 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:48.547102 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:48.553586 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:48.725631 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:48.801028 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:48.835491 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:48.838340 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:48.870864 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:48.905198 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:48.974566 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:49.270922 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:49.284310 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:49.294417 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:49.294705 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:49.464560 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:49.816606 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:49.842859 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:49.854357 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:49.867939 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:49.885790 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:49.981745 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:50.021863 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:50.078652 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:50.157411 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:50.207737 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:50.237695 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:50.359532 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:50.369676 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:50.375200 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:50.588344 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:50.755561 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:50.780781 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:50.785823 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:50.796080 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:50.839375 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:50.860661 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:50.876530 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:50.911445 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:51.011246 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:51.190538 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:51.293336 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:51.394522 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:51.415990 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:51.511737 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:51.539834 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:51.598201 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:51.664144 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:51.741477 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:51.825815 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:51.867314 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:52.047816 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:52.115753 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:52.231554 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:52.395277 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:52.438789 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:01:52.439632 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:52.480808 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:52.486114 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:52.517372 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:52.560777 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:52.580919 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:52.693480 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:52.717347 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:52.741300 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:52.804396 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:52.948372 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:01:52.951282 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:52.966078 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:52.991933 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:53.112075 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:01:53.148015 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:53.199306 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:53.231235 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:01:53.300647 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:53.370712 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:53.407809 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:53.444553 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:53.473511 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:53.514081 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:53.537719 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:53.619141 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:53.804520 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:53.828884 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:53.906585 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:54.000715 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:54.020215 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:54.025104 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:54.136488 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:54.152781 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:54.195607 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:54.303385 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:54.429955 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:54.603281 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:54.700070 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:54.803912 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:54.914463 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:54.921123 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:54.989379 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:55.118798 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:55.235960 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:55.293303 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:55.310148 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:55.320776 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:55.392716 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:55.395975 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:55.439894 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:55.475068 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:55.590010 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:55.706444 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:55.753528 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:55.816272 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:55.874588 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:55.923774 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:55.926339 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:55.975486 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:56.058709 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:56.134418 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:56.198918 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:56.324657 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:56.392374 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:56.463387 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:56.546088 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:56.580341 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:56.583153 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:56.688977 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:56.789515 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:56.804445 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:56.890273 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:57.019549 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:57.126066 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:57.207322 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:57.257108 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:57.321635 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:57.358880 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:57.368120 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:57.451756 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:01:57.467420 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:57.486434 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:57.692147 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:57.730049 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:57.758724 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:57.774801 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:57.781222 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:57.882014 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:57.918835 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:57.999028 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:58.115366 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:58.203341 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 55 similar messages]
W20240501 08:01:58.234872 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:58.248675 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 08:01:58.261714 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:58.277762 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:01:58.382874 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:58.399237 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:58.402436 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:58.572908 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:58.667901 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:58.680979 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:58.765178 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:58.821650 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:58.824671 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:58.854069 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:58.953567 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:58.976898 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:59.142774 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:59.232704 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:59.358801 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:59.368444 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:59.402901 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:59.677966 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:59.694005 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:59.812083 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:01:59.848204 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:59.885807 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:01:59.941112 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:59.975963 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:59.984328 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:00.229542 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:00.267422 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:02:00.277680 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:02:00.295467 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:02:00.305886 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:02:00.364830 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:00.415570 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:00.523481 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:00.552503 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:00.615728 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:00.673889 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:00.922519 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:00.970211 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:01.100193 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:02:01.101071 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:02:01.124269 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:01.128285 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:01.178272 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:02:01.198261 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:02:01.473031 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:02:01.490733 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:02:01.528769 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:01.529500 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:01.615857 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:01.834226 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:02:01.937781 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:01.985128 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:02.072896 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:02.122987 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:02.123329 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:02.204373 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:02:02.336341 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:02.472771 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:02:02.473615 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:02.530953 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:02.543367 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:02.692621 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:02.768864 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:02.776693 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:02.861549 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:02.904250 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:02.937276 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:02.990242 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:03.063046 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:03.095408 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:03.112432 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:03.198418 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:03.212132 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:02:03.306574 7164 maintenance_manager.cc:419] P 6aa238de40d94e5f9898ac424a4d1f68: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033369
I20240501 08:02:03.330446 7163 maintenance_manager.cc:643] P 6aa238de40d94e5f9898ac424a4d1f68: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.023s user 0.013s sys 0.010s Metrics: {"bytes_written":20869,"cfile_init":1,"dirs.queue_time_us":321,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1027,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":768,"lbm_writes_lt_1ms":29,"peak_mem_usage":0,"rows_written":43,"thread_start_us":143,"threads_started":2,"wal-append.queue_time_us":135}
I20240501 08:02:03.331454 7164 maintenance_manager.cc:419] P 6aa238de40d94e5f9898ac424a4d1f68: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 11545 bytes on disk
I20240501 08:02:03.332039 7163 maintenance_manager.cc:643] P 6aa238de40d94e5f9898ac424a4d1f68: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240501 08:02:03.352481 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:02:03.353186 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:03.375613 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:02:03.376333 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:03.547185 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:03.588352 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:02:03.603842 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:03.628631 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:03.629416 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:03.666915 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:03.783653 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:03.845752 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:03.911339 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:04.004616 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:04.085173 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:04.272106 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:04.375360 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:04.459539 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:04.529122 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:04.557736 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:04.700902 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:04.747938 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:02:04.886852 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:04.893039 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:04.958724 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:04.966691 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:05.047948 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:05.139431 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:05.146550 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:05.209828 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:05.408488 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:05.449189 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:05.459313 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:05.592643 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:05.647670 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:05.696738 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:05.707830 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:05.740491 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:05.831645 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:05.930619 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:05.933688 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:06.028759 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:06.029466 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:06.119460 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:06.127991 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:06.156890 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:06.171444 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:06.334353 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:06.450016 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:06.522287 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:06.658385 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:06.741376 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:06.814837 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:06.856433 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:06.983652 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:07.003783 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:07.063305 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:07.295780 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:07.344410 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:07.370301 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:02:07.381661 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:07.386994 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:07.530844 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:02:07.615209 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:07.666507 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:07.735445 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:07.776346 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:07.811336 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:07.908953 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:07.953541 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:07.976681 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:08.027354 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:08.135664 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:08.206892 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:08.263375 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:08.309525 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:08.315724 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:08.365418 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:08.367638 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:02:08.368248 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:08.433140 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:02:08.470332 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:08.599530 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:02:08.600327 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:02:08.651659 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling FlushMRSOp(3abb5202d8d24d9b9a67f7a87b7300a6): perf score=0.033334
I20240501 08:02:08.684664 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: FlushMRSOp(3abb5202d8d24d9b9a67f7a87b7300a6) complete. Timing: real 0.033s user 0.022s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":167,"dirs.queue_time_us":151,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":964,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":631,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":201,"threads_started":3,"wal-append.queue_time_us":165}
W20240501 08:02:08.686000 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:02:08.686039 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling UndoDeltaBlockGCOp(3abb5202d8d24d9b9a67f7a87b7300a6): 4586 bytes on disk
I20240501 08:02:08.686808 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: UndoDeltaBlockGCOp(3abb5202d8d24d9b9a67f7a87b7300a6) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240501 08:02:08.709959 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling FlushMRSOp(b8fdc2b9c2ef44b0bfec59a1a2f6b626): perf score=0.033353
W20240501 08:02:08.731873 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:02:08.738749 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(b357c740aca144b7b063845726f419e2): perf score=0.033362
W20240501 08:02:08.743310 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:02:08.756770 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: FlushMRSOp(b8fdc2b9c2ef44b0bfec59a1a2f6b626) complete. Timing: real 0.046s user 0.033s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":176,"dirs.queue_time_us":9216,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1055,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":795,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":269,"threads_started":3,"wal-append.queue_time_us":9604}
I20240501 08:02:08.758080 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling UndoDeltaBlockGCOp(b8fdc2b9c2ef44b0bfec59a1a2f6b626): 4926 bytes on disk
I20240501 08:02:08.758706 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: UndoDeltaBlockGCOp(b8fdc2b9c2ef44b0bfec59a1a2f6b626) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240501 08:02:08.759337 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling FlushMRSOp(b357c740aca144b7b063845726f419e2): perf score=0.033365
W20240501 08:02:08.763367 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:08.768122 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:08.782385 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: FlushMRSOp(b357c740aca144b7b063845726f419e2) complete. Timing: real 0.023s user 0.020s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1326,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":597,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":81,"threads_started":1,"wal-append.queue_time_us":386}
I20240501 08:02:08.783185 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling UndoDeltaBlockGCOp(b357c740aca144b7b063845726f419e2): 4839 bytes on disk
I20240501 08:02:08.783627 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: UndoDeltaBlockGCOp(b357c740aca144b7b063845726f419e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 08:02:08.792414 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(b357c740aca144b7b063845726f419e2) complete. Timing: real 0.053s user 0.030s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":7668,"dirs.queue_time_us":1810,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":2125,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":576,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":259,"threads_started":3,"wal-append.queue_time_us":192}
I20240501 08:02:08.794152 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(b357c740aca144b7b063845726f419e2): 4839 bytes on disk
I20240501 08:02:08.794786 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(b357c740aca144b7b063845726f419e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240501 08:02:08.868750 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling FlushMRSOp(b8fdc2b9c2ef44b0bfec59a1a2f6b626): perf score=0.033395
W20240501 08:02:08.911701 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:08.915910 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: FlushMRSOp(b8fdc2b9c2ef44b0bfec59a1a2f6b626) complete. Timing: real 0.047s user 0.033s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":11865,"dirs.queue_time_us":9540,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":2400,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":853,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":292,"threads_started":3,"wal-append.queue_time_us":7565}
I20240501 08:02:08.918061 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling UndoDeltaBlockGCOp(b8fdc2b9c2ef44b0bfec59a1a2f6b626): 4926 bytes on disk
I20240501 08:02:08.918774 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: UndoDeltaBlockGCOp(b8fdc2b9c2ef44b0bfec59a1a2f6b626) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
W20240501 08:02:09.130306 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:09.138516 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:09.274331 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:02:09.297281 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(21d76484f5dd499fb6fc3d534bffb779): perf score=0.033360
I20240501 08:02:09.309492 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(21d76484f5dd499fb6fc3d534bffb779) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":190,"dirs.run_cpu_time_us":290,"dirs.run_wall_time_us":1567,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":471,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":157,"threads_started":2,"wal-append.queue_time_us":154}
I20240501 08:02:09.310485 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(bde42e485c4b49ce8acd694a25ba683a): perf score=0.033361
W20240501 08:02:09.320492 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:09.324479 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(bde42e485c4b49ce8acd694a25ba683a) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1387,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1374,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":520,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":69,"threads_started":1,"wal-append.queue_time_us":152}
I20240501 08:02:09.325613 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(bde42e485c4b49ce8acd694a25ba683a): 4837 bytes on disk
I20240501 08:02:09.326112 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(bde42e485c4b49ce8acd694a25ba683a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240501 08:02:09.326961 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(21d76484f5dd499fb6fc3d534bffb779): 4586 bytes on disk
I20240501 08:02:09.327318 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(21d76484f5dd499fb6fc3d534bffb779) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240501 08:02:09.347152 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:09.421492 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling FlushMRSOp(54ced80b9ff943ec88c27b7842500b3d): perf score=0.033393
I20240501 08:02:09.436941 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: FlushMRSOp(54ced80b9ff943ec88c27b7842500b3d) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3626,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":562,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":150,"threads_started":2,"wal-append.queue_time_us":431}
I20240501 08:02:09.437971 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling FlushMRSOp(21d76484f5dd499fb6fc3d534bffb779): perf score=0.033397
I20240501 08:02:09.439731 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling FlushMRSOp(54ced80b9ff943ec88c27b7842500b3d): perf score=0.033399
W20240501 08:02:09.447520 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:02:09.451439 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: FlushMRSOp(21d76484f5dd499fb6fc3d534bffb779) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1329,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":581,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":104,"threads_started":1,"wal-append.queue_time_us":178}
I20240501 08:02:09.452209 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling UndoDeltaBlockGCOp(54ced80b9ff943ec88c27b7842500b3d): 4925 bytes on disk
I20240501 08:02:09.452595 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: UndoDeltaBlockGCOp(54ced80b9ff943ec88c27b7842500b3d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 08:02:09.453220 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling UndoDeltaBlockGCOp(21d76484f5dd499fb6fc3d534bffb779): 4586 bytes on disk
I20240501 08:02:09.453536 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: UndoDeltaBlockGCOp(21d76484f5dd499fb6fc3d534bffb779) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240501 08:02:09.453828 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: FlushMRSOp(54ced80b9ff943ec88c27b7842500b3d) complete. Timing: real 0.014s user 0.008s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":196,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1276,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":428,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":183,"threads_started":2,"wal-append.queue_time_us":146}
I20240501 08:02:09.454548 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling UndoDeltaBlockGCOp(54ced80b9ff943ec88c27b7842500b3d): 4925 bytes on disk
I20240501 08:02:09.454881 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: UndoDeltaBlockGCOp(54ced80b9ff943ec88c27b7842500b3d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240501 08:02:09.706202 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling FlushMRSOp(3d8ae3c82c9d4c5c858fa3f9d5959eb7): perf score=0.033338
I20240501 08:02:09.718608 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: FlushMRSOp(3d8ae3c82c9d4c5c858fa3f9d5959eb7) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1283,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":434,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":72,"threads_started":1,"wal-append.queue_time_us":152}
I20240501 08:02:09.719309 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling UndoDeltaBlockGCOp(3d8ae3c82c9d4c5c858fa3f9d5959eb7): 4926 bytes on disk
I20240501 08:02:09.719669 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: UndoDeltaBlockGCOp(3d8ae3c82c9d4c5c858fa3f9d5959eb7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240501 08:02:09.762143 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:09.786679 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling FlushMRSOp(3d8ae3c82c9d4c5c858fa3f9d5959eb7): perf score=0.033360
W20240501 08:02:09.792280 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:02:09.793373 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:02:09.799986 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: FlushMRSOp(3d8ae3c82c9d4c5c858fa3f9d5959eb7) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":161,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1355,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":410,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":149,"threads_started":2,"wal-append.queue_time_us":152}
I20240501 08:02:09.800792 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling FlushMRSOp(5970772c02d04e0b95a2f2feb7dcd655): perf score=0.033361
I20240501 08:02:09.813141 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: FlushMRSOp(5970772c02d04e0b95a2f2feb7dcd655) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1427,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":389,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":96,"threads_started":1,"wal-append.queue_time_us":185}
I20240501 08:02:09.814059 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling UndoDeltaBlockGCOp(3d8ae3c82c9d4c5c858fa3f9d5959eb7): 4926 bytes on disk
I20240501 08:02:09.814481 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: UndoDeltaBlockGCOp(3d8ae3c82c9d4c5c858fa3f9d5959eb7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240501 08:02:09.815212 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling UndoDeltaBlockGCOp(5970772c02d04e0b95a2f2feb7dcd655): 4839 bytes on disk
I20240501 08:02:09.815580 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: UndoDeltaBlockGCOp(5970772c02d04e0b95a2f2feb7dcd655) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240501 08:02:09.829806 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(cc6f7dcc930e4937865d37cb3c3bb3d2): perf score=0.033372
I20240501 08:02:09.841580 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(cc6f7dcc930e4937865d37cb3c3bb3d2) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":188,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1446,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":439,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":147,"threads_started":2,"wal-append.queue_time_us":177}
I20240501 08:02:09.842567 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(cc6f7dcc930e4937865d37cb3c3bb3d2): 4586 bytes on disk
I20240501 08:02:09.842929 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(cc6f7dcc930e4937865d37cb3c3bb3d2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240501 08:02:09.876713 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:09.905850 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:09.949909 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:10.152531 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:10.182044 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:02:10.206418 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling FlushMRSOp(5a37e631ae4f46a2be1ad5c43db5724b): perf score=0.033346
I20240501 08:02:10.219135 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: FlushMRSOp(5a37e631ae4f46a2be1ad5c43db5724b) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":207,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1312,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":475,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":175,"threads_started":2,"wal-append.queue_time_us":153}
I20240501 08:02:10.219957 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling UndoDeltaBlockGCOp(5a37e631ae4f46a2be1ad5c43db5724b): 4839 bytes on disk
I20240501 08:02:10.220330 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: UndoDeltaBlockGCOp(5a37e631ae4f46a2be1ad5c43db5724b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240501 08:02:10.221433 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling FlushMRSOp(1088a116ccec4020aec562a69791a188): perf score=0.033352
W20240501 08:02:10.227705 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:02:10.233045 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: FlushMRSOp(1088a116ccec4020aec562a69791a188) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":173,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1223,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":383,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":158}
I20240501 08:02:10.233745 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling UndoDeltaBlockGCOp(1088a116ccec4020aec562a69791a188): 4926 bytes on disk
I20240501 08:02:10.234166 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: UndoDeltaBlockGCOp(1088a116ccec4020aec562a69791a188) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240501 08:02:10.304527 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:10.317633 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling FlushMRSOp(1088a116ccec4020aec562a69791a188): perf score=0.033382
I20240501 08:02:10.330901 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: FlushMRSOp(1088a116ccec4020aec562a69791a188) complete. Timing: real 0.013s user 0.008s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1437,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":550,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":144,"threads_started":2,"wal-append.queue_time_us":149}
I20240501 08:02:10.331602 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling FlushMRSOp(0e42ef8df4884bc9bc33f5ed4c164c8c): perf score=0.033381
I20240501 08:02:10.345396 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(0e42ef8df4884bc9bc33f5ed4c164c8c): perf score=0.033386
I20240501 08:02:10.346261 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: FlushMRSOp(0e42ef8df4884bc9bc33f5ed4c164c8c) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1457,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1045,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":102,"threads_started":1,"wal-append.queue_time_us":389}
I20240501 08:02:10.348600 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling UndoDeltaBlockGCOp(1088a116ccec4020aec562a69791a188): 4926 bytes on disk
I20240501 08:02:10.349108 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: UndoDeltaBlockGCOp(1088a116ccec4020aec562a69791a188) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240501 08:02:10.350224 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:10.351477 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling UndoDeltaBlockGCOp(0e42ef8df4884bc9bc33f5ed4c164c8c): 4586 bytes on disk
I20240501 08:02:10.351891 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: UndoDeltaBlockGCOp(0e42ef8df4884bc9bc33f5ed4c164c8c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240501 08:02:10.352147 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:10.356335 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:10.357755 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:02:10.366537 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(0e42ef8df4884bc9bc33f5ed4c164c8c) complete. Timing: real 0.021s user 0.015s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":173,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":1458,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":558,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":144}
I20240501 08:02:10.367553 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(5a37e631ae4f46a2be1ad5c43db5724b): perf score=0.033391
I20240501 08:02:10.382928 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(5a37e631ae4f46a2be1ad5c43db5724b) complete. Timing: real 0.015s user 0.011s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":2437,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":523,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":75,"threads_started":1,"wal-append.queue_time_us":182}
I20240501 08:02:10.383826 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(5a37e631ae4f46a2be1ad5c43db5724b): 4839 bytes on disk
I20240501 08:02:10.384227 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(5a37e631ae4f46a2be1ad5c43db5724b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240501 08:02:10.385017 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(0e42ef8df4884bc9bc33f5ed4c164c8c): 4586 bytes on disk
I20240501 08:02:10.385376 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(0e42ef8df4884bc9bc33f5ed4c164c8c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
W20240501 08:02:10.444311 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:10.510735 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:10.549842 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:10.563212 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:10.713027 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:10.722226 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:10.887269 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:10.887835 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(7377810e08104526b8b8acdf886a3368): perf score=0.033350
W20240501 08:02:10.899574 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:10.900243 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(7377810e08104526b8b8acdf886a3368) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":162,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":1440,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":399,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":148}
I20240501 08:02:10.901115 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(7377810e08104526b8b8acdf886a3368): 4926 bytes on disk
I20240501 08:02:10.901446 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(7377810e08104526b8b8acdf886a3368) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240501 08:02:10.933352 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:10.946897 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:10.973166 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling FlushMRSOp(07967f68a412411494c6dd1f47209009): perf score=0.033372
I20240501 08:02:10.985181 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: FlushMRSOp(07967f68a412411494c6dd1f47209009) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":230,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":1348,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":494,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":138,"threads_started":2,"wal-append.queue_time_us":127}
I20240501 08:02:10.986073 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling UndoDeltaBlockGCOp(07967f68a412411494c6dd1f47209009): 4586 bytes on disk
I20240501 08:02:10.986495 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: UndoDeltaBlockGCOp(07967f68a412411494c6dd1f47209009) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240501 08:02:10.986539 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling FlushMRSOp(07967f68a412411494c6dd1f47209009): perf score=0.033373
I20240501 08:02:10.987206 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling FlushMRSOp(4dde9710a5eb4baa881d9dd04affe184): perf score=0.033374
I20240501 08:02:10.998605 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: FlushMRSOp(07967f68a412411494c6dd1f47209009) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":165,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1271,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":429,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":138,"threads_started":2,"wal-append.queue_time_us":159}
I20240501 08:02:10.999023 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: FlushMRSOp(4dde9710a5eb4baa881d9dd04affe184) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1363,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":405,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":62,"threads_started":1,"wal-append.queue_time_us":141}
I20240501 08:02:10.999423 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling UndoDeltaBlockGCOp(07967f68a412411494c6dd1f47209009): 4586 bytes on disk
I20240501 08:02:10.999758 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: UndoDeltaBlockGCOp(07967f68a412411494c6dd1f47209009) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240501 08:02:10.999804 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling UndoDeltaBlockGCOp(4dde9710a5eb4baa881d9dd04affe184): 4839 bytes on disk
I20240501 08:02:11.000151 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: UndoDeltaBlockGCOp(4dde9710a5eb4baa881d9dd04affe184) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240501 08:02:11.036010 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:11.142200 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:11.232117 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:11.251169 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling FlushMRSOp(dd17760cd5c2454db97610b2f71c0ee2): perf score=0.033353
I20240501 08:02:11.251636 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling FlushMRSOp(81ccdd855b584d65860fd7a5777920a3): perf score=0.033350
I20240501 08:02:11.268764 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: FlushMRSOp(dd17760cd5c2454db97610b2f71c0ee2) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":1326,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":619,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":78,"threads_started":1,"wal-append.queue_time_us":172}
I20240501 08:02:11.269352 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: FlushMRSOp(81ccdd855b584d65860fd7a5777920a3) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1159,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":537,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":69,"threads_started":1,"wal-append.queue_time_us":128}
I20240501 08:02:11.269644 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling UndoDeltaBlockGCOp(dd17760cd5c2454db97610b2f71c0ee2): 4586 bytes on disk
I20240501 08:02:11.270045 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: UndoDeltaBlockGCOp(dd17760cd5c2454db97610b2f71c0ee2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240501 08:02:11.270236 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling UndoDeltaBlockGCOp(81ccdd855b584d65860fd7a5777920a3): 4926 bytes on disk
I20240501 08:02:11.270617 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: UndoDeltaBlockGCOp(81ccdd855b584d65860fd7a5777920a3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240501 08:02:11.271320 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling FlushMRSOp(dd17760cd5c2454db97610b2f71c0ee2): perf score=0.033354
I20240501 08:02:11.286923 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: FlushMRSOp(dd17760cd5c2454db97610b2f71c0ee2) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":89,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":1373,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":590,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":57,"threads_started":1,"wal-append.queue_time_us":137}
I20240501 08:02:11.288057 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling UndoDeltaBlockGCOp(dd17760cd5c2454db97610b2f71c0ee2): 4586 bytes on disk
I20240501 08:02:11.288455 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: UndoDeltaBlockGCOp(dd17760cd5c2454db97610b2f71c0ee2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240501 08:02:11.323419 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:11.326951 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:02:11.357645 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling FlushMRSOp(e178f7c3c17c4e1ca3d16b6441f76e77): perf score=0.033380
I20240501 08:02:11.369329 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: FlushMRSOp(e178f7c3c17c4e1ca3d16b6441f76e77) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":191,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1385,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":416,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":152}
I20240501 08:02:11.370113 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling UndoDeltaBlockGCOp(e178f7c3c17c4e1ca3d16b6441f76e77): 4839 bytes on disk
I20240501 08:02:11.370455 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: UndoDeltaBlockGCOp(e178f7c3c17c4e1ca3d16b6441f76e77) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240501 08:02:11.395604 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:11.427953 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:11.436889 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:11.638605 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:11.841486 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:11.872300 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling FlushMRSOp(02b42b5fd3d440ea950b35ed461b8e12): perf score=0.033341
W20240501 08:02:11.875500 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:02:11.886723 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: FlushMRSOp(02b42b5fd3d440ea950b35ed461b8e12) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":194,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1417,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":586,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":122,"threads_started":2,"wal-append.queue_time_us":129}
I20240501 08:02:11.887442 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling UndoDeltaBlockGCOp(02b42b5fd3d440ea950b35ed461b8e12): 4926 bytes on disk
I20240501 08:02:11.887756 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: UndoDeltaBlockGCOp(02b42b5fd3d440ea950b35ed461b8e12) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240501 08:02:11.905222 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(8311d0921e9e417090e441884615525a): perf score=0.033352
I20240501 08:02:11.917029 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(8311d0921e9e417090e441884615525a) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":197,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1492,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":479,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":155,"threads_started":2,"wal-append.queue_time_us":152}
I20240501 08:02:11.917857 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(8311d0921e9e417090e441884615525a): 4586 bytes on disk
I20240501 08:02:11.918282 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(8311d0921e9e417090e441884615525a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
W20240501 08:02:12.006733 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:12.017527 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:12.022650 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling FlushMRSOp(4ea0db42a75a4879b3d3d608666dc2bf): perf score=0.033381
I20240501 08:02:12.035043 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: FlushMRSOp(4ea0db42a75a4879b3d3d608666dc2bf) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":180,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1424,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":440,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":145}
I20240501 08:02:12.035831 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling UndoDeltaBlockGCOp(4ea0db42a75a4879b3d3d608666dc2bf): 4839 bytes on disk
I20240501 08:02:12.036252 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: UndoDeltaBlockGCOp(4ea0db42a75a4879b3d3d608666dc2bf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 08:02:12.041412 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling FlushMRSOp(4ea0db42a75a4879b3d3d608666dc2bf): perf score=0.033386
I20240501 08:02:12.053335 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: FlushMRSOp(4ea0db42a75a4879b3d3d608666dc2bf) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":157,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1331,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":407,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":144,"threads_started":2,"wal-append.queue_time_us":138}
I20240501 08:02:12.054260 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling UndoDeltaBlockGCOp(4ea0db42a75a4879b3d3d608666dc2bf): 4839 bytes on disk
I20240501 08:02:12.054742 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: UndoDeltaBlockGCOp(4ea0db42a75a4879b3d3d608666dc2bf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240501 08:02:12.386935 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:12.411682 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:12.438709 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:12.442296 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:12.452775 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:12.476516 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:12.494861 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:12.538157 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling FlushMRSOp(8882528b26974c7aa73bb0ff0c444836): perf score=0.033339
I20240501 08:02:12.551244 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: FlushMRSOp(8882528b26974c7aa73bb0ff0c444836) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":220,"dirs.run_cpu_time_us":280,"dirs.run_wall_time_us":1338,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":551,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":157,"threads_started":2,"wal-append.queue_time_us":169}
I20240501 08:02:12.552044 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling UndoDeltaBlockGCOp(8882528b26974c7aa73bb0ff0c444836): 4926 bytes on disk
I20240501 08:02:12.552428 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: UndoDeltaBlockGCOp(8882528b26974c7aa73bb0ff0c444836) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 08:02:12.553015 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling FlushMRSOp(f647cfc562794e65a05918b241014d72): perf score=0.033340
I20240501 08:02:12.556946 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling FlushMRSOp(df6dd3d8482d4931b1d3014a67e739c6): perf score=0.033345
I20240501 08:02:12.571125 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: FlushMRSOp(f647cfc562794e65a05918b241014d72) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":1477,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":522,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":90,"threads_started":1,"wal-append.queue_time_us":2609}
I20240501 08:02:12.571889 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling UndoDeltaBlockGCOp(f647cfc562794e65a05918b241014d72): 4586 bytes on disk
I20240501 08:02:12.572320 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: UndoDeltaBlockGCOp(f647cfc562794e65a05918b241014d72) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 08:02:12.573666 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: FlushMRSOp(df6dd3d8482d4931b1d3014a67e739c6) complete. Timing: real 0.016s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":167,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1205,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":473,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":149}
I20240501 08:02:12.574472 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling UndoDeltaBlockGCOp(df6dd3d8482d4931b1d3014a67e739c6): 4839 bytes on disk
I20240501 08:02:12.574851 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: UndoDeltaBlockGCOp(df6dd3d8482d4931b1d3014a67e739c6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240501 08:02:12.669807 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:12.670804 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:12.671525 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(8882528b26974c7aa73bb0ff0c444836): perf score=0.033375
I20240501 08:02:12.687856 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(8882528b26974c7aa73bb0ff0c444836) complete. Timing: real 0.016s user 0.012s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":210,"dirs.run_cpu_time_us":297,"dirs.run_wall_time_us":1436,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":557,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":166,"threads_started":2,"wal-append.queue_time_us":164}
I20240501 08:02:12.689142 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(8882528b26974c7aa73bb0ff0c444836): 4926 bytes on disk
I20240501 08:02:12.689713 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(8882528b26974c7aa73bb0ff0c444836) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 08:02:12.690635 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(f647cfc562794e65a05918b241014d72): perf score=0.033379
I20240501 08:02:12.702589 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(f647cfc562794e65a05918b241014d72) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":88,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1456,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":446,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":73,"threads_started":1,"wal-append.queue_time_us":151}
I20240501 08:02:12.703544 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(f647cfc562794e65a05918b241014d72): 4586 bytes on disk
I20240501 08:02:12.703892 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(f647cfc562794e65a05918b241014d72) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240501 08:02:12.763363 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:02:12.868018 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:12.877945 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:12.923823 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:12.955369 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(cd07a00b6817400f86bf8c32bd592d8b): perf score=0.033334
I20240501 08:02:12.967610 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(cd07a00b6817400f86bf8c32bd592d8b) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1471,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":450,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":99,"threads_started":1,"wal-append.queue_time_us":187}
I20240501 08:02:12.968518 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(cd07a00b6817400f86bf8c32bd592d8b): 4926 bytes on disk
I20240501 08:02:12.968892 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(cd07a00b6817400f86bf8c32bd592d8b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240501 08:02:12.969580 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(5d18a2918c75467a9fe59979ce8936cd): perf score=0.033337
I20240501 08:02:12.981333 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(5d18a2918c75467a9fe59979ce8936cd) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1310,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":440,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":104,"threads_started":1,"wal-append.queue_time_us":181}
I20240501 08:02:12.982242 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(5d18a2918c75467a9fe59979ce8936cd): 4839 bytes on disk
I20240501 08:02:12.982607 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(5d18a2918c75467a9fe59979ce8936cd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240501 08:02:12.998512 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:13.023049 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:13.043479 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:13.056172 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:13.070466 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:13.074352 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling FlushMRSOp(5d18a2918c75467a9fe59979ce8936cd): perf score=0.033363
I20240501 08:02:13.077042 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling FlushMRSOp(ffc23756b44d4d588e0a8175075d9bc6): perf score=0.033367
I20240501 08:02:13.092211 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: FlushMRSOp(5d18a2918c75467a9fe59979ce8936cd) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":198,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1358,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":567,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":158,"threads_started":2,"wal-append.queue_time_us":138}
I20240501 08:02:13.092921 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling UndoDeltaBlockGCOp(5d18a2918c75467a9fe59979ce8936cd): 4839 bytes on disk
I20240501 08:02:13.093290 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: UndoDeltaBlockGCOp(5d18a2918c75467a9fe59979ce8936cd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240501 08:02:13.093380 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: FlushMRSOp(ffc23756b44d4d588e0a8175075d9bc6) complete. Timing: real 0.016s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":169,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1169,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":549,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":128,"threads_started":2,"wal-append.queue_time_us":118}
I20240501 08:02:13.094214 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling UndoDeltaBlockGCOp(ffc23756b44d4d588e0a8175075d9bc6): 4586 bytes on disk
I20240501 08:02:13.094645 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: UndoDeltaBlockGCOp(ffc23756b44d4d588e0a8175075d9bc6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240501 08:02:13.141381 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling FlushMRSOp(cd07a00b6817400f86bf8c32bd592d8b): perf score=0.033384
I20240501 08:02:13.156266 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: FlushMRSOp(cd07a00b6817400f86bf8c32bd592d8b) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":151,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":1258,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":457,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":203}
I20240501 08:02:13.157089 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling UndoDeltaBlockGCOp(cd07a00b6817400f86bf8c32bd592d8b): 4926 bytes on disk
I20240501 08:02:13.157719 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: UndoDeltaBlockGCOp(cd07a00b6817400f86bf8c32bd592d8b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240501 08:02:13.307363 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:13.328788 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:13.386544 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:13.409817 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:13.496892 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:02:13.497678 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:13.497949 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:02:13.508847 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:13.606817 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:02:13.623293 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:13.648924 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:13.735383 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(28baea6d7d3f442ba1dbfc5b37de90ac): perf score=0.033360
I20240501 08:02:13.747913 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(28baea6d7d3f442ba1dbfc5b37de90ac) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":184,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1365,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":396,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":143,"threads_started":2,"wal-append.queue_time_us":156}
I20240501 08:02:13.748852 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(28baea6d7d3f442ba1dbfc5b37de90ac): 4926 bytes on disk
I20240501 08:02:13.749332 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(28baea6d7d3f442ba1dbfc5b37de90ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240501 08:02:13.750258 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(4297f95243ac431099a8279c3d9e4b25): perf score=0.033363
W20240501 08:02:13.759444 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:13.763164 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(4297f95243ac431099a8279c3d9e4b25) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1191,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":427,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":81,"threads_started":1,"wal-append.queue_time_us":156}
I20240501 08:02:13.764045 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(4297f95243ac431099a8279c3d9e4b25): 4839 bytes on disk
I20240501 08:02:13.764436 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(4297f95243ac431099a8279c3d9e4b25) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240501 08:02:13.781147 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:13.845947 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling FlushMRSOp(4297f95243ac431099a8279c3d9e4b25): perf score=0.033390
I20240501 08:02:13.847923 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling FlushMRSOp(28baea6d7d3f442ba1dbfc5b37de90ac): perf score=0.033393
I20240501 08:02:13.864253 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: FlushMRSOp(4297f95243ac431099a8279c3d9e4b25) complete. Timing: real 0.018s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":658,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1387,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":593,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":157,"threads_started":2,"wal-append.queue_time_us":144}
I20240501 08:02:13.864997 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling UndoDeltaBlockGCOp(4297f95243ac431099a8279c3d9e4b25): 4839 bytes on disk
I20240501 08:02:13.865360 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: UndoDeltaBlockGCOp(4297f95243ac431099a8279c3d9e4b25) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240501 08:02:13.865813 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: FlushMRSOp(28baea6d7d3f442ba1dbfc5b37de90ac) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":139,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1129,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":533,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":123,"threads_started":2,"wal-append.queue_time_us":595}
I20240501 08:02:13.866772 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling FlushMRSOp(1e44704a9b9c46cf954d8e6937983841): perf score=0.033397
I20240501 08:02:13.878234 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: FlushMRSOp(1e44704a9b9c46cf954d8e6937983841) complete. Timing: real 0.011s user 0.008s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":1256,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":400,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":77,"threads_started":1,"wal-append.queue_time_us":163}
I20240501 08:02:13.879249 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling UndoDeltaBlockGCOp(28baea6d7d3f442ba1dbfc5b37de90ac): 4926 bytes on disk
I20240501 08:02:13.879685 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: UndoDeltaBlockGCOp(28baea6d7d3f442ba1dbfc5b37de90ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 08:02:13.880344 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling UndoDeltaBlockGCOp(1e44704a9b9c46cf954d8e6937983841): 4586 bytes on disk
I20240501 08:02:13.880671 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: UndoDeltaBlockGCOp(1e44704a9b9c46cf954d8e6937983841) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240501 08:02:13.936515 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:14.010361 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:14.080960 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:14.139838 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:14.221827 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:14.324167 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:14.556689 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:14.607581 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:14.739509 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:14.792843 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:14.929859 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:14.986896 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:15.035094 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:15.052903 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:15.197600 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:15.202821 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:15.231552 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:15.276708 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:15.424417 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:15.526707 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:15.541242 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:15.584980 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:15.603567 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:15.615617 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:15.649912 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:15.657884 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:15.725780 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:15.755755 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:15.759469 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:15.996730 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:16.029000 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:16.056174 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:16.060832 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:16.088934 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:16.255971 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:16.292718 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:16.468963 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:16.473285 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:16.515959 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:16.670183 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:16.700201 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:16.896239 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:16.983390 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:17.017066 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:17.112349 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:17.164520 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling FlushMRSOp(54ced80b9ff943ec88c27b7842500b3d): perf score=0.033364
I20240501 08:02:17.195974 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: FlushMRSOp(54ced80b9ff943ec88c27b7842500b3d) complete. Timing: real 0.031s user 0.020s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":226,"dirs.queue_time_us":598,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":927,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":535,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":286,"threads_started":3,"wal-append.queue_time_us":195}
I20240501 08:02:17.197409 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling UndoDeltaBlockGCOp(54ced80b9ff943ec88c27b7842500b3d): 4925 bytes on disk
I20240501 08:02:17.197949 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: UndoDeltaBlockGCOp(54ced80b9ff943ec88c27b7842500b3d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
W20240501 08:02:17.237612 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:17.276564 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:17.294137 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:17.403815 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:17.502544 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:17.603983 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:17.700073 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling FlushMRSOp(3abb5202d8d24d9b9a67f7a87b7300a6): perf score=0.033354
W20240501 08:02:17.701653 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:17.711726 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: FlushMRSOp(3abb5202d8d24d9b9a67f7a87b7300a6) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":204,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1328,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":438,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":155}
I20240501 08:02:17.712496 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling UndoDeltaBlockGCOp(3abb5202d8d24d9b9a67f7a87b7300a6): 4586 bytes on disk
I20240501 08:02:17.712813 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: UndoDeltaBlockGCOp(3abb5202d8d24d9b9a67f7a87b7300a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240501 08:02:17.744781 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:17.767748 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240501 08:02:17.788641 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling FlushMRSOp(5970772c02d04e0b95a2f2feb7dcd655): perf score=0.033348
W20240501 08:02:17.803905 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:02:17.815244 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: FlushMRSOp(5970772c02d04e0b95a2f2feb7dcd655) complete. Timing: real 0.026s user 0.021s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":174,"dirs.queue_time_us":156,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":806,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":450,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":247,"threads_started":3,"wal-append.queue_time_us":192}
I20240501 08:02:17.816562 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling UndoDeltaBlockGCOp(5970772c02d04e0b95a2f2feb7dcd655): 4839 bytes on disk
I20240501 08:02:17.817152 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: UndoDeltaBlockGCOp(5970772c02d04e0b95a2f2feb7dcd655) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240501 08:02:17.817843 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling FlushMRSOp(3abb5202d8d24d9b9a67f7a87b7300a6): perf score=0.033355
I20240501 08:02:17.837430 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: FlushMRSOp(3abb5202d8d24d9b9a67f7a87b7300a6) complete. Timing: real 0.019s user 0.015s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1378,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":554,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":92,"threads_started":1,"wal-append.queue_time_us":184}
I20240501 08:02:17.838382 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling UndoDeltaBlockGCOp(3abb5202d8d24d9b9a67f7a87b7300a6): 4586 bytes on disk
I20240501 08:02:17.838791 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: UndoDeltaBlockGCOp(3abb5202d8d24d9b9a67f7a87b7300a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240501 08:02:17.853360 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:17.923126 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:17.926347 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:17.960919 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:17.964174 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling FlushMRSOp(e178f7c3c17c4e1ca3d16b6441f76e77): perf score=0.033384
I20240501 08:02:17.982223 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: FlushMRSOp(e178f7c3c17c4e1ca3d16b6441f76e77) complete. Timing: real 0.018s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":86,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":1602,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":561,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":91,"threads_started":1,"wal-append.queue_time_us":176}
I20240501 08:02:17.983176 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling FlushMRSOp(4dde9710a5eb4baa881d9dd04affe184): perf score=0.033386
I20240501 08:02:17.999945 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: FlushMRSOp(4dde9710a5eb4baa881d9dd04affe184) complete. Timing: real 0.016s user 0.012s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":494,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1331,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":548,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":121,"threads_started":1,"wal-append.queue_time_us":221}
I20240501 08:02:18.001003 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling UndoDeltaBlockGCOp(4dde9710a5eb4baa881d9dd04affe184): 4839 bytes on disk
I20240501 08:02:18.001515 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: UndoDeltaBlockGCOp(4dde9710a5eb4baa881d9dd04affe184) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240501 08:02:18.002497 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling UndoDeltaBlockGCOp(e178f7c3c17c4e1ca3d16b6441f76e77): 4839 bytes on disk
I20240501 08:02:18.002851 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: UndoDeltaBlockGCOp(e178f7c3c17c4e1ca3d16b6441f76e77) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240501 08:02:18.005612 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:18.038643 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:18.067337 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:18.096482 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:18.254513 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling FlushMRSOp(1e44704a9b9c46cf954d8e6937983841): perf score=0.033372
I20240501 08:02:18.266332 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: FlushMRSOp(1e44704a9b9c46cf954d8e6937983841) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":1375,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":424,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":66,"threads_started":1,"wal-append.queue_time_us":146}
I20240501 08:02:18.267100 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling UndoDeltaBlockGCOp(1e44704a9b9c46cf954d8e6937983841): 4586 bytes on disk
I20240501 08:02:18.267499 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: UndoDeltaBlockGCOp(1e44704a9b9c46cf954d8e6937983841) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240501 08:02:18.268117 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling FlushMRSOp(cc6f7dcc930e4937865d37cb3c3bb3d2): perf score=0.033370
I20240501 08:02:18.279477 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: FlushMRSOp(cc6f7dcc930e4937865d37cb3c3bb3d2) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1428,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":401,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":96,"threads_started":1,"wal-append.queue_time_us":185}
I20240501 08:02:18.280310 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling FlushMRSOp(8311d0921e9e417090e441884615525a): perf score=0.033340
W20240501 08:02:18.291621 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:18.292835 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: FlushMRSOp(8311d0921e9e417090e441884615525a) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1444,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":466,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":117,"threads_started":1,"wal-append.queue_time_us":204}
I20240501 08:02:18.293601 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling UndoDeltaBlockGCOp(8311d0921e9e417090e441884615525a): 4586 bytes on disk
I20240501 08:02:18.293954 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: UndoDeltaBlockGCOp(8311d0921e9e417090e441884615525a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240501 08:02:18.294756 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling UndoDeltaBlockGCOp(cc6f7dcc930e4937865d37cb3c3bb3d2): 4586 bytes on disk
I20240501 08:02:18.295159 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: UndoDeltaBlockGCOp(cc6f7dcc930e4937865d37cb3c3bb3d2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240501 08:02:18.340917 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling FlushMRSOp(bde42e485c4b49ce8acd694a25ba683a): perf score=0.033363
W20240501 08:02:18.349529 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:18.357872 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: FlushMRSOp(bde42e485c4b49ce8acd694a25ba683a) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":172,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1584,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":515,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":155,"threads_started":2,"wal-append.queue_time_us":167}
I20240501 08:02:18.358719 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling UndoDeltaBlockGCOp(bde42e485c4b49ce8acd694a25ba683a): 4837 bytes on disk
I20240501 08:02:18.359113 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: UndoDeltaBlockGCOp(bde42e485c4b49ce8acd694a25ba683a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240501 08:02:18.359804 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling FlushMRSOp(81ccdd855b584d65860fd7a5777920a3): perf score=0.033367
I20240501 08:02:18.372810 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: FlushMRSOp(81ccdd855b584d65860fd7a5777920a3) complete. Timing: real 0.013s user 0.008s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1303,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":477,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":102,"threads_started":1,"wal-append.queue_time_us":187}
I20240501 08:02:18.373610 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling UndoDeltaBlockGCOp(81ccdd855b584d65860fd7a5777920a3): 4926 bytes on disk
I20240501 08:02:18.374095 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: UndoDeltaBlockGCOp(81ccdd855b584d65860fd7a5777920a3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240501 08:02:18.374886 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling FlushMRSOp(df6dd3d8482d4931b1d3014a67e739c6): perf score=0.033343
W20240501 08:02:18.375610 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:02:18.387286 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: FlushMRSOp(df6dd3d8482d4931b1d3014a67e739c6) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1242,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":403,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":108,"threads_started":1,"wal-append.queue_time_us":223}
I20240501 08:02:18.388027 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling UndoDeltaBlockGCOp(df6dd3d8482d4931b1d3014a67e739c6): 4839 bytes on disk
I20240501 08:02:18.388420 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: UndoDeltaBlockGCOp(df6dd3d8482d4931b1d3014a67e739c6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240501 08:02:18.389161 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling FlushMRSOp(1088a116ccec4020aec562a69791a188): perf score=0.033347
W20240501 08:02:18.390988 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:18.400858 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: FlushMRSOp(1088a116ccec4020aec562a69791a188) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1221,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":421,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":2816,"thread_start_us":107,"threads_started":1,"wal-append.queue_time_us":182}
I20240501 08:02:18.401580 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling UndoDeltaBlockGCOp(1088a116ccec4020aec562a69791a188): 4926 bytes on disk
I20240501 08:02:18.401911 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: UndoDeltaBlockGCOp(1088a116ccec4020aec562a69791a188) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240501 08:02:18.444491 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:18.489151 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:18.546041 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:02:18.546557 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling FlushMRSOp(02b42b5fd3d440ea950b35ed461b8e12): perf score=0.033389
I20240501 08:02:18.559347 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: FlushMRSOp(02b42b5fd3d440ea950b35ed461b8e12) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":1332,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":441,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":101,"threads_started":1,"wal-append.queue_time_us":262}
I20240501 08:02:18.560307 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling UndoDeltaBlockGCOp(02b42b5fd3d440ea950b35ed461b8e12): 4926 bytes on disk
I20240501 08:02:18.560761 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: UndoDeltaBlockGCOp(02b42b5fd3d440ea950b35ed461b8e12) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240501 08:02:18.561487 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling FlushMRSOp(b357c740aca144b7b063845726f419e2): perf score=0.033393
I20240501 08:02:18.573220 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: FlushMRSOp(b357c740aca144b7b063845726f419e2) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":70,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1393,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":441,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":116,"threads_started":1,"wal-append.queue_time_us":230}
I20240501 08:02:18.574160 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling UndoDeltaBlockGCOp(b357c740aca144b7b063845726f419e2): 4839 bytes on disk
I20240501 08:02:18.575307 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: UndoDeltaBlockGCOp(b357c740aca144b7b063845726f419e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240501 08:02:18.642899 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:02:18.643579 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:18.653138 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling FlushMRSOp(0e42ef8df4884bc9bc33f5ed4c164c8c): perf score=0.033384
I20240501 08:02:18.665705 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: FlushMRSOp(0e42ef8df4884bc9bc33f5ed4c164c8c) complete. Timing: real 0.012s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1509,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":473,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":95,"threads_started":1,"wal-append.queue_time_us":190}
I20240501 08:02:18.666548 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling UndoDeltaBlockGCOp(0e42ef8df4884bc9bc33f5ed4c164c8c): 4586 bytes on disk
I20240501 08:02:18.666886 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: UndoDeltaBlockGCOp(0e42ef8df4884bc9bc33f5ed4c164c8c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
W20240501 08:02:18.713743 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:18.713886 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:18.737648 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:18.791133 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:02:18.838548 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:18.911347 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:18.919785 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:19.025558 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:19.168874 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling FlushMRSOp(5d18a2918c75467a9fe59979ce8936cd): perf score=0.033355
I20240501 08:02:19.183640 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: FlushMRSOp(5d18a2918c75467a9fe59979ce8936cd) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":142,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":1767,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":495,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":123,"threads_started":2,"wal-append.queue_time_us":172}
I20240501 08:02:19.184423 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling UndoDeltaBlockGCOp(5d18a2918c75467a9fe59979ce8936cd): 4839 bytes on disk
I20240501 08:02:19.185102 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: UndoDeltaBlockGCOp(5d18a2918c75467a9fe59979ce8936cd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
W20240501 08:02:19.245790 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:19.258764 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:19.451563 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:19.578649 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling FlushMRSOp(3d8ae3c82c9d4c5c858fa3f9d5959eb7): perf score=0.033386
W20240501 08:02:19.580920 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:19.590765 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: FlushMRSOp(3d8ae3c82c9d4c5c858fa3f9d5959eb7) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":181,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1292,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":472,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":150,"threads_started":2,"wal-append.queue_time_us":140}
I20240501 08:02:19.591552 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling UndoDeltaBlockGCOp(3d8ae3c82c9d4c5c858fa3f9d5959eb7): 4926 bytes on disk
I20240501 08:02:19.591898 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: UndoDeltaBlockGCOp(3d8ae3c82c9d4c5c858fa3f9d5959eb7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240501 08:02:19.596585 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:19.643608 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:19.775955 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:19.863008 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:19.938047 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling FlushMRSOp(8311d0921e9e417090e441884615525a): perf score=0.033382
I20240501 08:02:19.948968 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: FlushMRSOp(8311d0921e9e417090e441884615525a) complete. Timing: real 0.011s user 0.009s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":194,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1419,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":420,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":138}
I20240501 08:02:19.949728 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling UndoDeltaBlockGCOp(8311d0921e9e417090e441884615525a): 4586 bytes on disk
I20240501 08:02:19.950098 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: UndoDeltaBlockGCOp(8311d0921e9e417090e441884615525a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240501 08:02:19.975699 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:19.976797 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:20.033929 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:20.154281 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:20.195883 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:20.199887 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:20.303305 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:20.344576 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling FlushMRSOp(ffc23756b44d4d588e0a8175075d9bc6): perf score=0.033368
I20240501 08:02:20.358681 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: FlushMRSOp(ffc23756b44d4d588e0a8175075d9bc6) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":175,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1457,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":541,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":135}
I20240501 08:02:20.359452 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling FlushMRSOp(8882528b26974c7aa73bb0ff0c444836): perf score=0.033338
I20240501 08:02:20.371392 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: FlushMRSOp(8882528b26974c7aa73bb0ff0c444836) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":1284,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":390,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":78,"threads_started":1,"wal-append.queue_time_us":164}
I20240501 08:02:20.372136 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling UndoDeltaBlockGCOp(8882528b26974c7aa73bb0ff0c444836): 4926 bytes on disk
I20240501 08:02:20.372502 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: UndoDeltaBlockGCOp(8882528b26974c7aa73bb0ff0c444836) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240501 08:02:20.373116 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling UndoDeltaBlockGCOp(ffc23756b44d4d588e0a8175075d9bc6): 4586 bytes on disk
I20240501 08:02:20.373426 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: UndoDeltaBlockGCOp(ffc23756b44d4d588e0a8175075d9bc6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240501 08:02:20.384394 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:20.431286 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:20.466297 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:20.512742 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:20.529009 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:20.575289 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:20.631651 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:20.650322 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling FlushMRSOp(5970772c02d04e0b95a2f2feb7dcd655): perf score=0.033394
I20240501 08:02:20.662694 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: FlushMRSOp(5970772c02d04e0b95a2f2feb7dcd655) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":167,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1434,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":472,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":175}
I20240501 08:02:20.663559 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling UndoDeltaBlockGCOp(5970772c02d04e0b95a2f2feb7dcd655): 4839 bytes on disk
I20240501 08:02:20.663908 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: UndoDeltaBlockGCOp(5970772c02d04e0b95a2f2feb7dcd655) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240501 08:02:20.702507 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling FlushMRSOp(7377810e08104526b8b8acdf886a3368): perf score=0.033334
I20240501 08:02:20.715056 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: FlushMRSOp(7377810e08104526b8b8acdf886a3368) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":1387,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":432,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":156,"threads_started":2,"wal-append.queue_time_us":174}
I20240501 08:02:20.715848 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling UndoDeltaBlockGCOp(7377810e08104526b8b8acdf886a3368): 4926 bytes on disk
I20240501 08:02:20.716225 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: UndoDeltaBlockGCOp(7377810e08104526b8b8acdf886a3368) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
W20240501 08:02:20.819273 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:20.831549 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:20.838425 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:20.865574 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:20.991870 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:21.004556 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:21.049541 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:21.060200 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:21.086382 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:02:21.125969 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling FlushMRSOp(df6dd3d8482d4931b1d3014a67e739c6): perf score=0.033349
I20240501 08:02:21.137935 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: FlushMRSOp(df6dd3d8482d4931b1d3014a67e739c6) complete. Timing: real 0.012s user 0.006s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":213,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":1558,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":424,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":160,"threads_started":2,"wal-append.queue_time_us":128}
I20240501 08:02:21.138835 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling UndoDeltaBlockGCOp(df6dd3d8482d4931b1d3014a67e739c6): 4839 bytes on disk
I20240501 08:02:21.139245 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: UndoDeltaBlockGCOp(df6dd3d8482d4931b1d3014a67e739c6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240501 08:02:21.218183 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling FlushMRSOp(21d76484f5dd499fb6fc3d534bffb779): perf score=0.033366
I20240501 08:02:21.230101 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: FlushMRSOp(21d76484f5dd499fb6fc3d534bffb779) complete. Timing: real 0.012s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":183,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1268,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":425,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":149,"threads_started":2,"wal-append.queue_time_us":130}
I20240501 08:02:21.230806 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling UndoDeltaBlockGCOp(21d76484f5dd499fb6fc3d534bffb779): 4586 bytes on disk
I20240501 08:02:21.231149 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: UndoDeltaBlockGCOp(21d76484f5dd499fb6fc3d534bffb779) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240501 08:02:21.312255 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:21.354525 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:21.377344 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:21.390483 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling FlushMRSOp(5a37e631ae4f46a2be1ad5c43db5724b): perf score=0.033343
I20240501 08:02:21.403064 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: FlushMRSOp(5a37e631ae4f46a2be1ad5c43db5724b) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1452,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":449,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":146}
I20240501 08:02:21.403829 9104 maintenance_manager.cc:419] P d8aa5ac376a54e1f8417e5af11d28fa7: Scheduling UndoDeltaBlockGCOp(5a37e631ae4f46a2be1ad5c43db5724b): 4839 bytes on disk
I20240501 08:02:21.404211 8994 maintenance_manager.cc:643] P d8aa5ac376a54e1f8417e5af11d28fa7: UndoDeltaBlockGCOp(5a37e631ae4f46a2be1ad5c43db5724b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240501 08:02:21.428905 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling FlushMRSOp(bde42e485c4b49ce8acd694a25ba683a): perf score=0.033366
W20240501 08:02:21.432628 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:21.441792 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: FlushMRSOp(bde42e485c4b49ce8acd694a25ba683a) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":189,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":1414,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":441,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":150}
I20240501 08:02:21.442509 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling UndoDeltaBlockGCOp(bde42e485c4b49ce8acd694a25ba683a): 4837 bytes on disk
I20240501 08:02:21.442843 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: UndoDeltaBlockGCOp(bde42e485c4b49ce8acd694a25ba683a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240501 08:02:21.547679 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:21.570160 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:21.790704 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(e178f7c3c17c4e1ca3d16b6441f76e77): perf score=0.033356
I20240501 08:02:21.802908 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(e178f7c3c17c4e1ca3d16b6441f76e77) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":172,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1624,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":396,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":128}
I20240501 08:02:21.803721 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(e178f7c3c17c4e1ca3d16b6441f76e77): 4839 bytes on disk
I20240501 08:02:21.804075 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(e178f7c3c17c4e1ca3d16b6441f76e77) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240501 08:02:21.830911 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:21.871737 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:21.886063 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:21.917955 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling FlushMRSOp(cc6f7dcc930e4937865d37cb3c3bb3d2): perf score=0.033346
I20240501 08:02:21.930172 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: FlushMRSOp(cc6f7dcc930e4937865d37cb3c3bb3d2) complete. Timing: real 0.012s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":185,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1374,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":429,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":168,"threads_started":2,"wal-append.queue_time_us":147}
I20240501 08:02:21.930963 7757 maintenance_manager.cc:419] P 09a7b6e2731a474b80411efb62323604: Scheduling UndoDeltaBlockGCOp(cc6f7dcc930e4937865d37cb3c3bb3d2): 4586 bytes on disk
I20240501 08:02:21.931416 7647 maintenance_manager.cc:643] P 09a7b6e2731a474b80411efb62323604: UndoDeltaBlockGCOp(cc6f7dcc930e4937865d37cb3c3bb3d2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240501 08:02:21.983784 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling FlushMRSOp(28baea6d7d3f442ba1dbfc5b37de90ac): perf score=0.033391
I20240501 08:02:21.995990 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: FlushMRSOp(28baea6d7d3f442ba1dbfc5b37de90ac) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":375,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1350,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":448,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":148,"threads_started":2,"wal-append.queue_time_us":150}
I20240501 08:02:21.996909 9366 maintenance_manager.cc:419] P 1c8cb9b1059a47cc822ca13a2b931776: Scheduling UndoDeltaBlockGCOp(28baea6d7d3f442ba1dbfc5b37de90ac): 4926 bytes on disk
I20240501 08:02:21.997288 9256 maintenance_manager.cc:643] P 1c8cb9b1059a47cc822ca13a2b931776: UndoDeltaBlockGCOp(28baea6d7d3f442ba1dbfc5b37de90ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240501 08:02:22.058348 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:22.216025 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:22.243762 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:22.306437 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(dd17760cd5c2454db97610b2f71c0ee2): perf score=0.033358
I20240501 08:02:22.317723 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(dd17760cd5c2454db97610b2f71c0ee2) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":182,"dirs.run_cpu_time_us":257,"dirs.run_wall_time_us":1527,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":414,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":139}
I20240501 08:02:22.318662 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(dd17760cd5c2454db97610b2f71c0ee2): 4586 bytes on disk
I20240501 08:02:22.319058 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(dd17760cd5c2454db97610b2f71c0ee2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240501 08:02:22.324077 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:22.340251 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:02:22.445959 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling FlushMRSOp(4dde9710a5eb4baa881d9dd04affe184): perf score=0.033396
I20240501 08:02:22.458401 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: FlushMRSOp(4dde9710a5eb4baa881d9dd04affe184) complete. Timing: real 0.012s user 0.005s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":191,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1500,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":421,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":135,"threads_started":2,"wal-append.queue_time_us":193}
I20240501 08:02:22.459074 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling UndoDeltaBlockGCOp(4dde9710a5eb4baa881d9dd04affe184): 4839 bytes on disk
I20240501 08:02:22.459439 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: UndoDeltaBlockGCOp(4dde9710a5eb4baa881d9dd04affe184) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
W20240501 08:02:22.489226 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:22.562657 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:02:22.570565 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling FlushMRSOp(b8fdc2b9c2ef44b0bfec59a1a2f6b626): perf score=0.033374
I20240501 08:02:22.584681 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: FlushMRSOp(b8fdc2b9c2ef44b0bfec59a1a2f6b626) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1317,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":525,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":117,"threads_started":1,"wal-append.queue_time_us":199}
I20240501 08:02:22.585515 7626 maintenance_manager.cc:419] P 31b02f06be8a4455b94b1254d77005bd: Scheduling UndoDeltaBlockGCOp(b8fdc2b9c2ef44b0bfec59a1a2f6b626): 4926 bytes on disk
I20240501 08:02:22.585939 7516 maintenance_manager.cc:643] P 31b02f06be8a4455b94b1254d77005bd: UndoDeltaBlockGCOp(b8fdc2b9c2ef44b0bfec59a1a2f6b626) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240501 08:02:22.763298 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:22.789469 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:22.796871 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:22.804129 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240501 08:02:22.804876 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:22.839133 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:22.892076 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:22.925305 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:22.942853 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:22.945060 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:23.006781 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:23.055985 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:23.172281 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:23.374419 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:23.433624 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:23.483089 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:23.513824 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:23.536525 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:23.584837 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:02:23.585498 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:23.620479 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:23.635150 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:23.675294 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:02:23.707706 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:23.862483 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:23.884313 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:23.898893 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:23.924327 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:24.009750 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 08:02:24.217186 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:24.311803 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:24.362860 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:24.516764 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:24.604408 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:24.658339 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:24.767706 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:24.787906 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:24.901713 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:24.945801 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:24.984038 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:25.211226 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:25.215687 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:25.239228 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:25.265244 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:25.342873 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:25.344933 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:25.347159 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:25.396247 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:25.400835 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:25.409080 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:25.641571 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:25.754184 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:25.805392 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:25.814415 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:25.830536 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:25.932508 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:25.936429 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:26.001708 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:26.109781 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:26.163985 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:26.302181 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:26.324433 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:26.340633 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:26.360889 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:26.402824 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:26.735283 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:26.750495 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:26.839514 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:27.001350 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:27.177345 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:27.308434 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:27.334566 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:27.383323 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:27.443102 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:27.481086 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:27.562698 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:27.598058 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:27.618425 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:27.619278 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5970772c02d04e0b95a2f2feb7dcd655: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:27.741936 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:27.749138 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:27.760386 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:27.771415 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:27.821949 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:27.843003 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:02:27.843892 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:27.883687 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:27.938673 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:27.938771 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:28.221172 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:28.288981 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:28.306605 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:28.377208 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:28.428205 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:28.492345 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:28.546599 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:28.659307 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:28.661070 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:02:28.667173 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:28.679611 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:28.789930 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:28.790028 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:02:28.793015 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:28.816208 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:28.898298 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:29.003855 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:29.013232 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 08:02:29.269416 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:29.280746 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:29.302654 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:29.530279 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:29.542555 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:29.676268 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:29.726043 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:29.841753 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:29.891373 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:29.994602 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:30.008008 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:30.054548 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:30.088011 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:30.123034 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:30.185142 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:30.199759 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:30.222537 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:30.224701 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:30.273116 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:30.293543 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:30.298424 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:30.455221 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:30.737465 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:30.862629 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:30.883538 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:30.906596 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:30.910679 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:30.918247 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:30.924038 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:30.971828 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:31.042366 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:31.156960 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:31.303387 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:31.350687 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:31.407526 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:31.423589 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:31.607933 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:31.785157 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:31.912113 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:31.919497 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:31.998019 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:32.186950 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:32.194912 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:32.321115 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:32.340224 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:32.442618 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:32.468955 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:32.504176 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:32.513460 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:32.548324 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:32.564658 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:32.644228 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:32.760468 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:32.810405 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:32.812940 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:32.886858 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:32.890128 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:02:32.925881 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:33.074294 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:33.198112 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:33.236559 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:33.316164 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:33.490504 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:33.521668 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:33.546619 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:33.629245 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:33.662822 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:02:33.697065 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:33.713829 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:33.734213 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:33.813288 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:33.866840 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240501 08:02:33.867435 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:33.977849 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:34.024580 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:02:34.182103 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:34.208794 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:34.292618 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:34.377465 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:34.621470 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:34.715716 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:34.759860 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:34.780105 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:34.834765 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:34.867825 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:34.878939 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:34.902442 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:34.967344 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:34.971530 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:35.011029 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:35.171528 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:35.199714 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:35.312130 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:35.333420 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:35.382694 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:35.421939 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:35.512508 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:35.670753 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:35.792369 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:35.797745 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:35.887995 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:35.917711 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:35.992888 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:36.131052 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:36.141021 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:36.240424 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:36.275735 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:36.337653 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:36.337895 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:36.478775 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:36.566128 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:36.694160 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:36.739857 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:36.833201 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:37.021788 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:37.128899 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:37.159173 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:37.275027 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:37.359434 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:37.366065 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:37.487934 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:37.497962 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:37.525943 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:37.583289 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:37.625000 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:37.631297 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:37.635569 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:37.660880 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:37.686777 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:37.714876 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:37.930614 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240501 08:02:38.032989 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:38.042644 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:38.200369 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:38.200501 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:38.265499 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:38.339123 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:38.496353 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:38.496588 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:38.569876 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:38.578676 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:38.724712 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:38.740893 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:02:38.741554 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:38.783115 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:38.901629 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:38.922775 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:38.994222 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:02:39.055588 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:02:39.112710 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:39.117013 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:39.151769 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:39.383947 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:39.425050 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:39.541867 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:39.567752 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:39.585124 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:39.728940 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:39.874470 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:39.892752 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:39.972316 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:40.019873 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:40.041612 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:40.095239 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:40.107976 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:40.117364 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:40.132061 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:40.269212 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:40.276963 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:40.419821 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:40.482141 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:40.601176 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:40.644114 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:40.683887 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:40.702921 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:40.959403 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:40.983817 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:41.026962 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:41.182652 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:41.187237 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:41.252249 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:41.384495 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:41.429399 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:41.453661 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:41.504482 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:41.530953 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:41.559266 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:41.679075 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:41.854934 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:41.888070 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:41.919392 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:42.120687 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:42.147070 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:42.186487 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:42.298347 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:42.337083 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:42.442446 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:42.472441 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:42.480058 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:42.517513 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:42.561784 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:42.619686 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:42.664579 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:42.838109 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:42.868530 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:42.876319 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:42.953744 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:02:43.016026 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:43.070914 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:43.079731 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:43.239306 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:43.265295 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:43.375684 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:43.392427 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:43.484782 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:43.581501 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:43.854836 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:43.887404 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:43.937705 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:02:43.968719 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:43.995977 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:44.031881 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:44.058652 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:02:44.059253 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:44.085152 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 08:02:44.133122 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:44.144346 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:44.256502 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:44.431937 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:44.465727 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:44.480114 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:44.565496 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:44.726009 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:44.751047 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:44.757316 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:44.842293 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:44.845398 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:44.846966 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:44.876155 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:44.925881 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:45.058112 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:45.129681 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:45.140808 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:45.157709 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:45.371624 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:45.412178 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:45.494549 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:45.585695 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:45.597846 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:45.602327 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:45.739737 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:45.940831 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:45.989934 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:46.208563 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:46.210050 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:46.222762 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:46.381513 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:46.395474 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:46.458802 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:46.680583 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:46.697851 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:46.730226 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:46.737129 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:46.815851 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:46.826774 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:46.933857 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:47.102635 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:47.139457 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:47.140784 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:47.195952 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:47.257663 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:47.439953 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:47.455824 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:47.499425 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:47.551987 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:47.591446 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:47.593917 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:47.609691 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:47.700826 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:47.734830 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:47.802686 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:47.989074 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:02:47.989797 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:48.008422 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:48.040524 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:48.171065 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:48.273499 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:48.310266 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:48.498567 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:48.567142 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:48.624161 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:48.692359 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:48.744134 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:48.775240 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:48.779597 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:48.955538 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:49.078565 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240501 08:02:49.079214 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:49.134158 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:02:49.134197 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:02:49.161038 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:49.371259 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:49.384786 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:49.418802 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:49.477362 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:49.622027 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:49.632135 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:49.664222 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:49.804950 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:49.828366 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:49.921835 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:49.935118 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:50.008771 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:50.043597 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:50.096096 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:50.119947 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:50.144966 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:50.154592 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:50.170996 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:50.329216 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:50.349083 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:50.427297 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:50.444307 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:50.449126 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:50.481130 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:50.542527 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:50.759429 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:50.850697 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:50.922840 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:51.007058 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:51.117919 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:51.236876 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:51.284634 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:51.385290 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:51.448375 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:51.449400 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:51.752089 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:51.945363 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:51.983280 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:52.025416 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:52.072007 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:52.157410 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:52.160362 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:52.177345 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:52.216099 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:52.317837 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:52.360208 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:52.385687 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:52.532155 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:52.607399 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:52.642488 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:52.658722 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:52.677565 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:52.729614 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:52.737079 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:52.781167 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:52.834559 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:52.882412 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:53.094043 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:53.106360 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:53.115557 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:02:53.274957 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:53.328150 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:53.330749 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:53.345882 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:53.348399 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:53.552512 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:53.680557 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:53.846371 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:53.934916 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:53.966643 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:53.988353 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:54.032864 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:54.119078 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:02:54.162158 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:02:54.257236 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 08:02:54.270185 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:54.352005 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:54.394150 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:54.468000 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:54.523926 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:54.571533 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:54.604140 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:54.619118 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:54.679441 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:54.752832 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:54.973722 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:55.083233 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:55.089344 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:55.101073 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:55.146178 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:55.184208 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:55.289855 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:55.294973 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:55.340400 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:55.364899 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:55.386435 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:55.644088 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:55.649152 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:55.693728 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:55.765792 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:55.793015 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:55.816962 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:56.001317 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:56.122794 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:56.235807 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:56.256878 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:56.302050 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:56.305796 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:56.417963 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:56.548594 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:56.560712 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:56.661293 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:56.751230 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:56.791507 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:56.926581 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:56.969606 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:57.067936 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:57.116677 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:57.175426 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:57.209965 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:57.222599 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:57.331332 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:57.386093 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:57.442296 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:57.483582 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:57.562502 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:57.588083 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:57.828177 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:57.856775 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:57.917778 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:57.974733 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:58.048136 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:58.082302 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:58.112453 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:58.134752 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:58.242906 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:58.261869 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:02:58.303901 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:58.356997 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:58.407166 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:58.478996 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:58.515712 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:58.527191 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:58.766526 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:58.890285 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:58.917701 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:58.948899 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:59.002440 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:59.060009 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:59.121840 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:59.192781 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:02:59.299482 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:02:59.299598 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:59.300186 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:59.428519 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:02:59.429317 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:59.444169 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:59.594280 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:59.603073 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:59.712246 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:59.787511 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:59.795818 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:59.834023 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:59.899621 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:02:59.984972 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:02:59.987810 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:03:00.060920 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:03:00.198479 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:00.340004 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:03:00.340868 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:00.373513 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:03:00.532294 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:00.591853 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:00.605238 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:03:00.608990 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:00.690109 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:03:00.709844 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:03:00.811200 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:03:00.818619 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:00.939188 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:00.995710 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:03:01.020830 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:01.177527 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:01.436170 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:01.490563 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:03:01.539873 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:01.568346 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:03:01.611136 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:01.700220 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:01.717954 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:03:01.719553 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:01.864370 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:01.877090 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:01.928469 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:02.150475 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:02.313329 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:02.337671 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:03:02.364467 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:02.427528 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:02.450457 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:02.455092 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:02.627698 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:02.656639 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:02.684988 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:02.705600 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:02.772529 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:02.834833 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:02.862226 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:03:02.943295 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:02.949812 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:03.180716 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:03.219645 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:03.219709 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:03.255370 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:03.274101 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:03:03.462726 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:03.474911 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:03.692831 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:03.760208 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:03:03.816543 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:03.820504 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:03.886904 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:04.125638 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:04.148020 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:04.204490 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:04.209470 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:03:04.222275 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:04.248349 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:04.259429 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:04.336732 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:03:04.337633 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:04.353955 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:04.366431 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:04.559231 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:03:04.666656 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:04.691783 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:04.838223 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:04.914028 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:04.916833 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:04.999262 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:05.111517 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:05.123715 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:05.135265 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:05.234968 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:05.252624 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:05.309799 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:05.386606 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:05.395151 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:05.422448 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:05.551175 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:05.640439 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:05.662672 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:05.835103 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:05.914614 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:05.972222 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:06.073256 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:03:06.203249 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:06.329774 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:06.332990 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:06.353547 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:06.355527 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:06.360978 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:06.588444 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:06.682556 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:06.778571 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:06.825260 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:06.841517 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:06.871352 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:06.874642 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:06.945241 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:06.999229 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:07.208742 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:07.310250 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:07.402910 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:07.440799 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:07.456907 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:07.462970 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:07.501400 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:07.595053 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:07.681351 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:07.777835 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:07.796361 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:07.843091 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:07.997001 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:08.036123 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:08.108129 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:08.133163 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:08.233157 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:08.271675 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:08.376813 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:03:08.447075 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:08.588881 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:08.652851 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:08.675572 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:03:08.680719 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:08.765642 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:08.916399 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:08.918592 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:08.931028 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:09.108558 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:09.232705 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:09.260618 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:03:09.339323 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:09.387530 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:09.398119 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:09.440371 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:09.487776 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:09.568303 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:09.622089 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:03:09.623117 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:09.667860 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:09.704053 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:03:09.784685 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:09.883343 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:09.970691 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:10.026144 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:10.065637 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:10.118841 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:10.161850 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:10.162001 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:10.287040 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:10.293609 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:10.347378 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:10.497960 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:10.529731 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:10.554054 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:10.584371 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:10.587420 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:10.792220 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:10.801198 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:03:10.836091 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:11.026284 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:11.084960 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:11.154287 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:11.261677 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:11.416556 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:11.435936 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:11.467401 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:11.502305 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:11.571897 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:11.591238 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:11.880759 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:11.894685 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:11.908471 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:11.982291 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:12.012413 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:12.074599 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:12.079466 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:12.162803 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:12.268539 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:12.285131 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:12.336165 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:12.526006 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:12.605429 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:12.646138 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:12.719426 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:12.764631 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:12.794397 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:12.840641 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:12.844959 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:12.888317 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:13.007004 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:13.018178 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:13.030920 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:13.099035 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:13.234150 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:13.295500 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:13.305629 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:03:13.405542 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:03:13.659394 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:13.701550 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:13.714151 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:13.773150 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:13.779402 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:13.779655 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:13.784078 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:14.051577 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:14.228015 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:14.272456 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:14.275240 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:14.305187 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:03:14.382876 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:14.413489 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:14.416991 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:14.448119 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:14.464752 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:14.573408 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:14.595249 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:14.622452 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240501 08:03:14.623199 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:14.844067 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:03:14.854841 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:14.932061 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:15.121343 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:15.136937 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:15.195082 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:15.335759 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:15.417524 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:15.428459 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:15.433404 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:15.449327 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:15.466557 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:15.519187 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:15.622727 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:15.708242 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:15.792610 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:15.835232 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:15.927278 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:16.043854 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:16.073109 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:16.091174 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:16.225229 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:16.295614 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:16.368271 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:16.442260 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:16.520944 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:16.585223 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:16.747736 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:16.836263 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:16.895141 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:16.899132 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:16.921952 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:17.029109 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:17.075248 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:17.117074 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:17.129683 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:17.148537 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:17.414040 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:17.450402 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:17.456046 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:17.595345 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:17.656216 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:17.853165 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:17.945083 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:17.961112 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:17.999615 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:18.040582 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:18.049401 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:18.076840 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:18.099459 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:18.123284 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:18.230011 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:18.285310 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:18.330667 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:18.525964 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240501 08:03:18.552474 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:18.641675 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:18.669611 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:18.684185 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:18.726963 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:18.746098 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:18.961073 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:19.008762 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:19.194034 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:19.242316 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:19.247087 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:19.279832 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:19.312913 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:19.345914 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:03:19.370672 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:19.406966 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:19.513283 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:19.556340 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:19.697878 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:19.710022 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:03:19.726691 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:19.742779 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:19.846555 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:19.920287 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:03:19.920966 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:20.061121 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:20.167377 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:20.307178 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:20.315423 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:20.524605 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:20.562944 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:20.652036 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:20.667155 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:20.749385 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:20.795626 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:20.847682 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:20.867034 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:20.890935 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:20.905815 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:20.969970 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:21.062101 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:21.126979 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:21.266727 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:21.374436 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:21.455744 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:21.564046 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:21.576108 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:21.584261 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:21.600817 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:21.632594 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:21.667591 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:21.752450 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:21.824787 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:21.866824 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:22.017371 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:22.019729 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:22.038336 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:22.167526 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:22.208098 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:22.363423 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:22.481952 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:22.493264 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:22.513919 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:22.734922 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:22.764787 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:22.993515 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:22.994386 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:23.006665 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:23.007761 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:23.123903 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:23.145653 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:23.207770 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:23.326169 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:23.458173 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:23.470412 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:23.521224 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:23.531719 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:23.557477 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:23.664850 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:03:23.704062 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:23.729566 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:23.750380 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:23.827353 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:23.880318 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:23.967952 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:24.082276 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:24.111434 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:24.146952 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:24.259390 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:24.276118 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:24.313697 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:24.351778 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:24.494501 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:03:24.521382 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:24.668220 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:24.692867 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:24.732564 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 08:03:24.733235 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:24.780164 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:24.858980 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:24.884399 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:25.139526 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:25.145150 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:03:25.190268 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:25.228529 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:25.345741 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:25.452600 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:25.598498 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:25.686051 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:25.690941 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:25.729380 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:25.741675 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:25.753033 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:25.795713 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:25.816783 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:25.986143 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:26.026867 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:26.101162 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:26.166961 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:26.185189 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:26.185719 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:26.208030 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:26.290856 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:26.527345 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:26.613231 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:26.624449 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:26.627492 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:26.711406 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:26.717761 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:26.847354 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:26.952862 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:27.160686 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:27.169459 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:27.217064 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:27.219108 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:27.399210 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:27.414072 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:27.474660 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:27.581923 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:27.646623 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:27.647413 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5970772c02d04e0b95a2f2feb7dcd655: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:27.709609 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:27.740053 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:27.843583 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:28.009721 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:28.084535 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:28.109882 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:28.134482 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:28.167008 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:28.170223 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:28.221755 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:28.278863 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:28.405827 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:28.409202 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:28.468895 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:28.576071 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:28.664602 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:28.683663 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:03:28.717844 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:28.735049 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:28.738816 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:28.790705 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:28.923149 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:29.045689 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:29.071005 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:29.196501 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:29.207298 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:29.254360 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:29.310303 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:29.333500 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:29.557695 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:29.565896 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:29.633756 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:03:29.715224 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:29.740067 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:03:29.752014 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:29.830277 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:29.937479 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:30.012334 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:30.139699 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:30.155648 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:03:30.205359 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:30.235481 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:30.262259 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:30.391067 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:30.498805 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:30.570623 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:30.630966 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:30.651118 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:30.695770 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:30.723639 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:30.748750 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:30.779474 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:30.882541 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:31.003095 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:31.062209 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:31.082393 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:31.198397 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:31.216560 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:31.315790 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:31.350764 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:31.409719 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:31.511071 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:31.513263 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:31.547322 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:31.574498 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:31.710628 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:31.775146 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:31.890861 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:31.918350 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:32.073177 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:32.082450 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:32.153832 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:32.154505 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:32.369037 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:32.583364 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:32.678375 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:32.730554 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:32.800354 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:32.931838 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:32.947911 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:33.020375 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:33.051918 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:33.115675 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:33.126854 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:33.196360 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:33.288390 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:33.298236 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:33.397240 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:33.426728 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:33.473419 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:33.556841 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:33.596469 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:33.658979 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:33.775369 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:33.794715 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:03:33.909771 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:33.912348 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:33.940567 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:34.023682 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:34.147934 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:34.153864 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:34.156177 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:34.183306 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:34.250304 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:34.304982 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:34.313885 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:34.624583 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:34.698530 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:34.713495 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:03:34.729795 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:34.746500 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:34.955201 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:35.046211 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 08:03:35.046949 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:35.161000 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:03:35.161703 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:35.174300 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:35.194892 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:35.205004 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:35.400368 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:35.411365 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:35.574273 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:35.585479 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:35.644990 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:35.666579 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:35.726579 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:35.727871 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:35.837244 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:35.866390 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:35.878711 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:35.958763 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:36.047257 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:36.107365 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:36.152946 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:36.238520 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:36.487432 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:36.517302 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:36.576102 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:36.601751 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:36.650192 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:36.685925 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:36.688946 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:36.699317 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:36.933912 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:36.966342 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:37.030999 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:37.111153 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:37.152756 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:37.264839 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:37.291721 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:37.383010 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:37.528882 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:37.532990 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:37.602182 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:37.818595 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:37.859194 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:38.108661 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:38.124325 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:38.137362 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:38.173453 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:38.226248 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:38.259409 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:38.280678 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:38.280771 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:38.343807 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:38.383014 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:38.491015 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:38.581629 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:38.789034 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:38.801292 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:03:38.814525 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:38.823621 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:38.907047 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:38.973130 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:39.032737 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:39.115780 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:39.252466 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:39.255631 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:39.268399 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:39.289426 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:39.312935 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:39.325778 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:39.331637 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:39.461580 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:39.634953 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:39.698427 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:39.717633 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:03:39.734392 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:39.795154 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:39.804603 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:39.986239 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:40.119424 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:03:40.149163 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:40.175463 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:40.203886 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 08:03:40.245079 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:40.414781 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:40.524056 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:40.570627 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:40.621399 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:40.649128 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:40.711011 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:40.719192 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:40.807454 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:40.821164 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:40.833377 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:40.849107 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:40.952675 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:41.210141 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:41.226688 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:41.352679 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:41.367754 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:41.390816 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:41.582494 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:41.584476 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:41.626390 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:41.636827 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:41.646169 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:41.745097 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:41.776082 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:41.781555 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:41.785881 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:41.833559 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:41.895377 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:41.922875 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:42.138847 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:42.164171 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:42.239387 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:42.542393 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:42.578071 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:42.584668 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:42.658052 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:42.772526 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:42.925093 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:43.020804 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:43.081945 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:43.092952 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:43.226191 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:43.251732 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:43.289304 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:43.350181 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:43.365859 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:43.461557 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:43.481434 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:43.654606 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:43.680735 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:43.736557 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:43.810778 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:03:43.898255 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:43.955503 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:43.964278 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:44.020977 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:44.052485 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:44.061336 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:44.141139 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:44.193096 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:44.224187 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:44.248595 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:44.336211 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:44.363085 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:44.369975 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:44.389953 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:44.571281 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:44.646596 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:44.744211 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:03:44.775434 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:45.029570 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:45.052464 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:45.093212 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:45.123605 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:03:45.124413 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:45.152526 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:45.314800 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:45.451143 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 08:03:45.499521 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:45.553582 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:45.573217 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:45.631175 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:45.739327 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:45.944334 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:45.980356 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:45.980582 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:46.046062 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:46.069583 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:46.085947 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:46.086858 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:46.163445 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:46.402678 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:46.436841 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:46.448880 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:46.507267 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:46.523801 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:46.543686 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:46.587699 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:46.654696 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:46.686053 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:46.757501 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:46.804379 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:46.827217 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:46.891222 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:46.913694 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:46.934231 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:47.051352 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:47.151072 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:47.379235 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:47.423930 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:47.432924 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:47.550005 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:47.554435 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:47.776072 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:47.949208 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:47.990003 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:48.023217 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:48.103389 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:48.112090 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:48.136596 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:48.498966 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:48.499255 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:48.536060 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:48.564139 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:48.599663 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:48.659086 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:48.676501 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:48.718195 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:48.957319 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:03:48.978729 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:48.988987 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:49.007149 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:49.028100 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:49.060204 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:49.128557 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:49.138338 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:49.187068 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:49.193357 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:49.227820 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:49.250524 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:49.358578 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:49.401273 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:49.467250 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:49.670079 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:49.708263 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:49.772204 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:03:49.857831 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:49.875988 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:49.946874 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:50.067979 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:50.124826 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:50.242238 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:03:50.265918 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:50.282570 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:50.286840 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:50.430888 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:50.463546 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:50.494362 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:03:50.669740 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:50.702703 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:50.726969 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:50.777056 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:51.031564 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:51.084239 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:51.084434 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:51.176698 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:51.210985 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:51.269023 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:51.304723 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:51.338789 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:51.392902 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:51.428615 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:51.462028 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:51.468276 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:51.471092 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:51.602483 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:51.646566 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:51.677904 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:51.684165 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:51.749869 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:51.794170 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:51.899974 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:52.055159 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:52.098591 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:52.210992 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:52.324631 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:52.368309 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:52.455978 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:52.525514 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:52.595806 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:52.662943 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:52.872397 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:52.945292 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:52.952045 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:52.970237 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:52.999689 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:53.078847 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:53.363384 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:53.456514 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:53.536298 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:53.575407 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:53.640273 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:53.732779 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:53.748392 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:53.758550 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:53.773396 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:53.793850 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:53.842815 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:53.888513 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:53.939028 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:53.982949 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:53.987463 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:54.000595 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:03:54.078136 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:54.176932 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:54.254228 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:54.340797 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:54.364023 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:54.533953 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:54.548491 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:54.601012 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:54.634428 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:54.874590 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:03:54.944243 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:55.003445 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:55.022380 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:55.028823 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:55.030323 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:55.093644 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:55.129680 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:55.323657 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 08:03:55.438498 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:55.544024 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:55.547540 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 08:03:55.567960 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:55.613812 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:55.631389 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:55.801851 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:55.952533 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:56.055554 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:56.131808 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:56.147368 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:56.190183 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:56.208916 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:56.224016 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:56.234195 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:56.304486 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:56.415408 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:56.417620 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:56.453812 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:56.542462 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:56.560252 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:56.603529 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:56.684453 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:56.699847 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:56.891359 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:56.900089 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:56.913669 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:56.984329 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:57.040844 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:57.080224 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:57.509786 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:57.525225 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:57.552863 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:57.571453 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:57.625811 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:57.636662 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:57.648039 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:57.862020 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:58.027052 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:58.050581 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:58.088014 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:58.128074 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:58.387514 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:58.390794 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:58.491885 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:58.581010 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:58.652199 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:58.667340 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:58.684738 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:58.741415 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:58.766265 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:58.780092 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:58.864423 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:58.928406 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:59.050971 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:59.092376 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:59.092775 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:03:59.097819 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:59.120052 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:59.193578 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:59.245574 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:59.286777 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:59.608753 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:03:59.621269 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:03:59.689462 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:59.753697 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:59.835381 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:59.861084 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:00.058393 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:04:00.099509 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:04:00.162797 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:00.167080 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:00.210136 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:04:00.247187 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:04:00.299407 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:00.434866 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:04:00.439471 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:04:00.657768 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:00.671728 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:04:00.672461 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:00.720562 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:00.721537 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:04:00.902825 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:04:00.987462 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:04:01.056695 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:01.117966 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:01.134140 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:01.162706 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:01.179138 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:01.183969 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:04:01.208287 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:01.280160 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:04:01.287298 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:01.293088 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:01.361290 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:04:01.572966 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:04:01.667927 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:01.685761 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:01.697418 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:01.842219 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:01.963858 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:04:02.065510 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:02.080732 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:02.207571 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:02.255203 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:02.315656 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:04:02.406350 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:02.518214 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:02.622444 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:02.677719 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:02.720036 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:02.755235 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:02.858614 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:02.867169 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:04:03.058164 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:03.099139 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:03.130336 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:03.247781 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:03.342736 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:03.472712 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:03.477854 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:03.515972 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:03.570780 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:03.591910 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:03.685786 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:03.694506 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:03.715521 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:03.719695 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:03.773713 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:03.831925 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:03.899044 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:03.920996 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:04.042681 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:04.100953 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:04:04.102280 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:04.267001 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:04.285861 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:04.495735 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:04.559958 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:04.607373 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:04.619777 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:04.820487 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:04.838800 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:04.923487 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:05.016582 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:05.112946 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:05.122057 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:04:05.178776 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:05.254973 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:04:05.270833 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:05.408339 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:05.409106 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:05.551465 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:04:05.602768 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:05.605238 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:05.718737 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:04:05.749365 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:05.781353 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:05.892931 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:05.993660 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:06.018538 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:06.158317 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:06.202718 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:06.203951 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:06.235248 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:06.259505 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:06.296643 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:06.299999 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:06.317524 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:06.412763 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:06.502450 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:06.586031 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:06.615013 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:06.856055 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:06.904278 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:06.985628 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:07.003641 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:07.064472 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:07.163250 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:07.291993 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:07.366629 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:07.447566 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:07.579367 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:07.582085 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:07.754228 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:07.762646 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:07.802862 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:07.838744 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:07.977339 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:04:08.041625 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:08.186511 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:08.188082 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:08.244730 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:08.301963 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:08.307230 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:08.556615 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:08.654785 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:08.737210 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:08.777247 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:08.804371 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:08.810611 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:08.826201 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:08.866429 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:08.870380 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:08.881939 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:09.045871 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:09.064899 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:09.137812 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:04:09.138446 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:09.164503 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:09.353930 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:09.354501 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:09.559556 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:09.583660 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:09.617064 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:09.669608 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:09.873366 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:09.908587 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:09.924510 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:09.947767 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:10.013375 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:10.251964 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:10.253698 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:04:10.321305 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:10.379374 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:10.383029 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:10.481034 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:10.497429 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:10.527601 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:10.551970 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:10.726876 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:04:10.844038 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:10.876590 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:04:10.877204 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:10.943781 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:11.014647 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:11.032630 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:11.113170 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:11.296937 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:11.299674 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:11.306995 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:11.308197 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:11.352785 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:11.378379 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:11.400259 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:11.462772 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:11.540340 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:11.596499 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:11.722851 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:11.795230 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:11.810149 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:11.962934 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:11.964727 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:12.174046 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:12.251744 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:12.278681 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:12.304423 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:12.432971 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:12.528306 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:12.668941 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:12.777082 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:12.839601 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:12.854529 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:12.894765 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:13.098515 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:13.177129 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:13.318897 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:13.349714 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:13.405714 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:13.424410 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:13.454344 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:13.474977 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:13.573240 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:13.638341 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:13.686841 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:13.740671 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:13.767760 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:13.884312 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:13.892568 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:13.984699 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:14.034333 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:14.039085 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:14.168843 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240501 08:04:14.194859 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:14.210992 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:14.257752 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:14.306354 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:14.444020 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:14.531173 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:14.545854 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:14.580487 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:14.721943 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:14.775317 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:14.876863 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:15.034147 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:15.093191 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:15.124948 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:15.221100 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:15.260449 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:04:15.486980 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:15.605620 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:15.627913 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:15.672757 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:15.714907 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:15.766705 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:15.815693 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:15.901086 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 08:04:15.902383 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:15.985612 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:16.044020 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:04:16.105043 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:16.126926 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:16.139678 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:16.157719 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:16.179179 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:16.252604 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:16.275647 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:16.290366 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:16.555986 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:16.631531 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:16.646203 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:16.684906 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:16.691022 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:16.869436 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:16.883396 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:16.957810 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:16.982517 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:17.069438 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:17.087534 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:17.134837 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:17.258625 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:17.270588 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:17.514868 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:17.667696 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:17.686475 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:17.717330 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:17.873098 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:17.903177 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:17.971426 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:18.071753 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:18.089092 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:18.238315 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:18.284147 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:18.338290 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:18.418908 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:18.459627 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:18.552592 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:18.582276 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:18.692193 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:18.735289 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:18.751263 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:18.793406 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:18.880900 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:18.894145 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:19.137391 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:19.178288 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:04:19.179131 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:19.216778 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:19.231269 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:19.280570 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:19.333135 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:19.385094 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:19.431546 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:19.440148 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:19.608299 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:19.643865 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:19.719440 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:19.836431 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:19.995057 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:20.120445 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:20.204031 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:20.297559 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:04:20.356436 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:20.408725 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:20.499814 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:20.523310 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:20.572129 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:20.690165 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:20.723938 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:20.742727 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:20.785276 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:20.785954 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:20.829845 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:21.066704 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:04:21.067478 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:21.067512 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:21.088068 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:04:21.088796 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:21.150034 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:21.169891 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:21.267334 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:21.288231 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:21.448604 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:21.474277 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:21.611444 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:21.653131 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:21.733322 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:21.770856 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:21.771565 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:21.772367 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:21.813467 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:22.050222 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:22.106317 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:22.156534 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:22.261392 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:22.295606 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:22.444041 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:22.538677 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:22.576413 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:22.750221 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:22.834934 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:22.912204 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:22.930773 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:23.044417 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:23.112504 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:23.326707 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:23.417168 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:23.418840 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:23.424659 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:23.453028 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:23.465667 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:23.497691 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:23.655016 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:23.657035 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:23.690673 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:23.754240 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:23.816022 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:23.911531 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:24.006019 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:24.045135 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:24.047597 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:24.125021 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:24.184054 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:04:24.286113 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:24.400602 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:24.493083 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:24.503300 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:24.547235 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:24.572541 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:24.629494 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:24.769007 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:24.868613 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:24.915341 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:24.988179 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:24.990126 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:25.015208 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:25.094869 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:25.257902 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:25.319787 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:04:25.474583 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:25.483386 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:25.493664 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:25.598383 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:25.712229 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:25.776047 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:25.880894 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:25.892992 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:25.930253 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:25.970772 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:26.014489 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:26.077656 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:26.136981 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:04:26.152899 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:26.198858 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:26.230823 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:26.279786 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 08:04:26.280458 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:26.316124 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:26.338059 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:26.497653 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:26.568360 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:26.706396 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:26.736425 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:26.739218 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:26.870625 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:26.920105 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:26.959466 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:26.966717 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:27.052341 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:27.267030 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:27.283397 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:27.324884 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:27.396286 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:27.416143 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:27.642602 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:27.689185 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:27.689882 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5970772c02d04e0b95a2f2feb7dcd655: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:27.718519 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:27.724380 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:27.811923 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:27.856828 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:28.153512 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:28.217185 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:28.262827 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:28.344880 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:28.357877 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:28.421828 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:28.432816 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:28.625154 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:28.626966 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:28.645416 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:28.648524 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:28.665966 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:28.790050 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:28.874982 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:28.918474 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:28.959023 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:28.996999 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:29.034296 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:29.036669 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:29.201445 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240501 08:04:29.303146 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:29.354292 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:29.417331 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:29.485397 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:29.543882 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:29.582008 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:29.626869 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:29.894949 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:29.904767 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:30.010191 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:30.039548 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:30.041523 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:30.061571 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:30.263250 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:30.337765 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:04:30.383144 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:30.391254 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:30.456265 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:30.524060 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:30.853106 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:30.869771 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:30.899821 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:30.900635 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:30.950893 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:31.007615 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:31.023499 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:31.083482 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:31.123090 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:31.135506 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:31.243937 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:31.339475 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:31.343997 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:31.359309 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:04:31.404162 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:31.463657 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:04:31.464300 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:31.467458 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:31.603349 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:31.629907 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:31.651288 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:32.003293 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:32.010632 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:32.021337 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:32.091428 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:32.175305 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:32.253747 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:32.371023 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:32.519604 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:32.531315 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:32.584333 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:32.626638 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:32.684677 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:32.851146 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:32.942821 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:33.008766 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:33.071533 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:33.182658 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:33.184962 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:33.278741 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:33.383620 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:33.478626 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:33.514883 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:33.564525 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:33.565371 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:33.581441 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:33.667337 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:33.708807 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:33.712112 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:33.797457 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:33.824920 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:33.917362 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:33.927192 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:34.118726 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:34.138348 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:34.163738 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:34.194653 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:34.260202 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:04:34.547474 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:34.553671 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:34.615202 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:34.756855 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:34.808111 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:34.923748 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:34.963208 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:35.066224 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:35.078207 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:35.124919 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:35.131989 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:35.200362 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:35.339124 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:04:35.428638 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:35.486166 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:35.598263 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:35.609068 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:35.637202 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:35.648690 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:35.889361 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:35.979513 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:35.985178 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:36.015815 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:36.055349 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:36.181191 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:36.238662 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:36.258841 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:36.273406 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:36.307953 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:36.310171 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:36.314247 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:36.325279 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:36.369015 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:36.450274 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:04:36.471616 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:04:36.584177 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:36.639179 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:36.766883 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:36.794353 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:37.024698 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:37.069864 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:37.086865 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:37.159432 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:37.326680 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:37.379261 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:37.528415 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:37.557132 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:37.581087 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:37.587728 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:37.715957 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:37.781296 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:37.882055 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:38.044075 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:38.109768 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:38.157521 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:38.170799 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:38.268872 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:38.297739 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:38.357652 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:38.439877 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:38.505399 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:38.579645 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:38.598795 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:38.643464 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:38.706763 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:38.742547 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:38.748824 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:38.794023 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:38.823079 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:38.985491 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:39.052619 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:39.062781 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:39.169281 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:39.263247 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:04:39.271682 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:39.321817 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:39.400949 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:39.639016 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:39.709005 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:39.738576 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:39.851358 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:39.960047 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:40.064153 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:40.153555 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:40.160171 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:40.232079 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:40.264036 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:40.282419 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:40.286368 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:40.389331 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:04:40.602865 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:40.612293 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:40.667150 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:40.769026 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:40.809023 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:40.824615 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:40.870748 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:40.902958 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:40.905381 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:41.069512 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:41.143059 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:41.147109 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:41.158051 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:41.249790 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:41.278316 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:41.303356 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:41.474057 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:04:41.497468 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:04:41.498183 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:41.622277 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:41.623114 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:41.652433 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:41.692306 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:41.915058 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:41.948554 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:42.067158 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:42.119354 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:42.280228 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:42.479002 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:42.493741 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:42.512820 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:42.526916 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:42.555280 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:42.560540 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:42.590442 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:42.683852 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:42.899883 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:43.074154 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:43.083573 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:43.125962 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:43.133630 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:43.191154 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:43.344743 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:43.461431 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:43.569867 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:43.572479 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:43.614475 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:43.656106 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:43.674723 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:43.678181 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:43.759212 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:43.785594 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:44.120915 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:44.158120 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:44.184175 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:44.191524 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:44.215063 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:44.296752 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:44.299407 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:44.495976 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:04:44.496767 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:44.646987 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:44.677271 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:44.806303 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:44.831346 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:44.914417 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:45.092264 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:45.100157 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:45.187732 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:45.212594 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:45.266026 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:45.275846 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:45.345198 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:45.441083 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:04:45.449023 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:45.533566 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:45.593309 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:45.754998 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:45.772640 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:45.804358 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:45.867811 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:45.874766 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:45.874855 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:46.065590 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:46.172785 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:46.183207 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:46.210312 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:46.293540 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:46.298584 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:46.339884 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:46.357434 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:46.483399 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:46.581211 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:04:46.614545 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:04:46.662029 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:46.752555 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:46.788332 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:46.934408 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:46.935099 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:47.184394 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:47.227561 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:47.326498 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:47.346416 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:47.349310 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:47.536805 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:47.565865 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:47.571605 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:47.633579 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:47.740087 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:47.759836 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:47.783792 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:47.823587 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:47.875102 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:47.912149 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:48.097376 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:48.277583 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:48.300588 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:48.303063 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:48.423184 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:48.523365 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:48.589257 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:48.612015 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:48.651445 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:48.724746 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:48.740339 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:48.784250 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:48.907344 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:48.965443 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:49.017832 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:49.027748 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:49.087553 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:49.168053 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:49.198333 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:49.297163 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:49.517190 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:04:49.550978 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:49.619551 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:49.665956 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:49.692902 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:49.767500 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:49.770294 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:49.847136 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:49.880715 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:50.052491 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:50.145644 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:50.160604 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:50.164191 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:50.210784 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:50.267597 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:50.420904 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:50.452684 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 08:04:50.653635 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:50.665349 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:50.709621 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:50.719995 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:50.746917 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:50.771782 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:50.925760 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:51.088620 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:51.182277 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:51.204255 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:51.227142 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:51.244375 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:51.263300 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:51.499123 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:51.518868 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:51.546216 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:51.556036 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:51.586750 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:04:51.627082 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:51.671794 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:51.700403 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:04:51.749693 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:51.789564 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:51.940660 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:51.950716 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:52.053766 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:52.180038 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:52.181322 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:52.189751 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:52.304548 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:52.325119 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:52.394022 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:52.430912 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:52.459110 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:52.677794 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:52.868877 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:52.887991 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:52.899572 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:52.901003 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:52.921247 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:52.925792 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:53.006631 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:53.256713 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:53.339121 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:53.363853 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:53.402513 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:53.536875 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:53.653767 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:53.818835 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:53.852044 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:53.855064 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:53.921166 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:53.933837 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:53.946966 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:54.086151 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:54.096957 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:54.118371 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:54.313299 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:54.325737 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:54.381294 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:54.529685 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:04:54.530354 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:54.544744 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:54.556191 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:54.633827 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:54.691008 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:54.782397 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:54.787678 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:54.814219 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:54.833138 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:55.074379 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:55.283633 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:55.323530 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:55.352958 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:55.374157 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:55.392333 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:55.520326 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:55.554206 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:04:55.554935 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:55.840226 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:55.843197 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:55.844478 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:55.887265 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:55.970758 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:56.118667 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:56.181879 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:56.206344 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:56.242722 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:56.344154 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:56.435431 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:56.439463 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:56.469592 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:56.534723 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:56.716035 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:04:56.716674 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:56.742714 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:56.752388 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:56.843937 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:56.896981 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:04:56.938339 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:56.948555 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:56.983733 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:57.023670 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:57.316974 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:57.365684 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:57.374085 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:57.385807 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:57.462608 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:57.498059 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:57.521950 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:57.707832 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:57.765560 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:57.786414 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:57.896332 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:57.957175 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:57.960932 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:58.136850 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:58.140504 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:04:58.181898 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:58.207118 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:58.282361 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:04:58.350507 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:58.401396 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:58.566044 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:04:58.591799 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:58.885643 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:58.900199 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:04:58.961483 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:58.988940 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:59.023102 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:59.133121 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:59.149576 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:59.182932 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:04:59.212519 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:04:59.230691 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:59.347103 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:59.376904 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:59.499282 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:59.524271 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:04:59.530728 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:04:59.593119 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:04:59.798100 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:04:59.879566 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:59.915230 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:59.982147 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:04:59.997790 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:00.083058 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:00.159397 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:05:00.199409 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:05:00.407104 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:00.426734 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:05:00.500367 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:00.500826 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:00.508502 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:00.590801 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:05:00.602589 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:05:00.795022 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:00.825131 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:05:00.860726 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:05:00.957304 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:00.965581 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:01.026899 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:01.113865 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:01.317385 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:05:01.339120 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:01.414824 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:01.496060 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:01.518641 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:01.607120 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:01.667294 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:05:01.668347 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:05:01.754423 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:05:01.762377 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:01.802709 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:01.802959 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:01.843490 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:01.891979 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:05:01.997422 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:02.011875 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:05:02.024813 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 08:05:02.211675 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:02.254626 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:02.275378 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:05:02.396059 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:02.477332 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:02.567797 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:02.592058 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:05:02.596817 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:02.635434 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:02.638530 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:05:02.799544 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:02.880602 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:02.983983 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:03.006035 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:03.024530 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:03.101320 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:03.208724 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:03.406576 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:03.409521 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:03.418679 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:03.492825 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:03.562561 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:03.917156 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:03.919791 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:03.930490 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:03.974432 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:03.991448 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:04.016547 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:04.044206 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:04.080685 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:04.140058 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:05:04.173378 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:04.436511 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:04.463703 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:04.511009 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:04.567437 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:05:04.647059 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:04.744329 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:04.798097 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:04.877874 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:04.888252 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:04.933144 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:05.094273 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:05.106267 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:05.120875 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:05.269376 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:05.270241 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:05.336014 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:05.482985 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:05.534215 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:05.550333 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:05.581388 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:05.735437 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:05.848925 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:05.881785 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:05.882952 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:05:05.891314 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:05.957585 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:06.020982 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:06.203516 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:06.212592 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:06.309425 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:06.362027 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:06.410051 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:06.410598 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:06.490065 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:06.535013 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:05:06.678177 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:06.701431 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:06.715378 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:06.890415 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:05:07.003659 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:07.083485 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:05:07.084158 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:07.152561 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:07.162277 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:07.314903 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:07.366912 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:07.489615 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:07.504207 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:07.531574 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:07.606192 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:07.646154 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:07.667523 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:07.687068 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:07.713557 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:07.946967 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:08.020635 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:08.036234 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:08.073058 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:08.117187 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:08.270700 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:08.285914 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:08.407997 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:08.476855 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:08.494678 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:08.496821 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:08.514874 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:08.705411 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:08.796871 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:08.833127 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:08.877461 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:08.901583 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:08.919013 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:05:08.920902 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:09.018390 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:09.185940 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:09.240976 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:09.335264 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:09.386397 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:09.407315 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:09.497083 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:09.589565 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:09.686216 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:05:09.842159 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:09.879249 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:09.966073 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:09.966176 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:09.968366 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:09.985774 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:10.130051 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:10.265957 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:10.324242 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:10.428804 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:10.433352 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:10.493049 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:10.551105 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:10.595969 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:10.726436 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:10.731105 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:10.785147 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:10.899035 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:10.919173 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:10.921303 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:05:10.978935 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:11.002811 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:11.054343 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:11.307898 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:11.309623 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:11.319432 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:05:11.407009 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:11.433537 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:11.456861 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:11.538439 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:11.643844 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:11.829939 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:11.875978 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:12.034693 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:12.056740 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:12.089313 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:05:12.112250 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:12.209004 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 08:05:12.256101 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:12.261407 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:12.273550 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:12.317257 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:12.349610 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:12.497903 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:12.513319 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:12.658398 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:12.680935 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:12.737504 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:12.811618 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:12.811594 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:12.868225 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:12.961578 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:13.006284 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:13.141063 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:13.183486 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:13.209390 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:13.245342 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:13.286799 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:13.607205 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:13.607527 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:13.628410 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:13.650722 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:13.752568 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:13.772287 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:13.803414 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:13.808714 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:13.811025 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:13.983609 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:14.087508 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:14.261086 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:14.401998 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:14.501188 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:14.609014 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:14.710367 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:14.754585 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:14.786792 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:14.797248 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:05:14.818374 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:14.875613 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:14.988991 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:15.041741 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:15.094694 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:15.102660 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:15.266885 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:15.335628 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:15.409483 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:15.429800 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:15.489497 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:15.574499 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:15.639225 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:15.662834 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:15.697213 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:15.787086 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:15.822039 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:16.007345 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:05:16.092185 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:16.176378 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:16.272740 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:16.276918 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:16.293370 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:16.338568 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:16.387794 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:16.391227 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:16.452644 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:16.473168 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:16.613337 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:16.758953 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:16.960362 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:16.978123 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:17.137099 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:17.140910 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:05:17.141593 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:17.250341 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:05:17.304400 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:17.350571 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:17.356000 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:17.384444 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:17.386868 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:17.547585 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:17.728123 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:17.771163 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:17.783545 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:17.809780 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:17.943092 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:17.976558 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:18.022276 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:18.040174 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:18.090318 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:18.101152 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:18.169622 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:18.364321 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:18.374812 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:18.420392 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:18.608279 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:18.676177 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:18.805541 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:18.849617 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:18.863006 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:18.864488 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:18.988075 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:18.989045 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:19.031980 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:19.077859 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:19.188452 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:19.375710 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:19.403694 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:19.412782 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:19.567081 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:19.598757 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:19.706024 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:19.783553 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 08:05:19.800048 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:05:19.816809 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:19.823357 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:19.992166 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:20.015183 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:20.146675 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:20.193284 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:20.231949 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:20.235954 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:20.461722 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:20.466476 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:20.547070 7251 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 08:05:20.563122 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:20.574950 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:20.620265 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:20.666369 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:20.669945 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:05:20.790352 13214 consensus_queue.cc:579] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Leader has been unable to successfully communicate with peer 3b69c5d799e14e438df760453438aa75 for more than 300 seconds (300.040s)
I20240501 08:05:20.796615 7709 consensus_queue.cc:237] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
W20240501 08:05:20.798517 7645 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81ccdd855b584d65860fd7a5777920a3. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:20.798884 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:20.799082 9318 raft_consensus.cc:1261] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:05:20.799407 14183 consensus_queue.cc:1035] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:05:20.801079 14186 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:05:20.801548 9318 raft_consensus.cc:2932] T 81ccdd855b584d65860fd7a5777920a3 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:05:20.803036 7162 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 81ccdd855b584d65860fd7a5777920a3 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:20.804365 7171 catalog_manager.cc:5484] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 11 to 12, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:05:20.817174 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:20.843107 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:20.907486 13825 consensus_queue.cc:579] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Leader has been unable to successfully communicate with peer 3b69c5d799e14e438df760453438aa75 for more than 300 seconds (300.359s)
W20240501 08:05:20.908924 7382 consensus_peers.cc:462] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:05:20.913717 7446 consensus_queue.cc:237] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:05:20.915508 7575 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:05:20.915567 9318 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:05:20.916152 14216 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:05:20.916364 13825 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:05:20.918525 13825 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:05:20.920472 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5d18a2918c75467a9fe59979ce8936cd with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:20.920982 7575 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:05:20.920955 9318 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:05:20.921259 7175 catalog_manager.cc:5484] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 7 to 8, NON_VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:05:20.924593 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:05:20.925172 7446 consensus_queue.cc:237] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
W20240501 08:05:20.925413 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5d18a2918c75467a9fe59979ce8936cd: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:05:20.926702 7575 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:20.927203 14216 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:05:20.927307 9318 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:20.927750 14216 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:05:20.928915 14215 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
I20240501 08:05:20.929447 7575 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
W20240501 08:05:20.930617 7384 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): Couldn't send request to peer b4f8020266b5455b872a2973fc396578. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d18a2918c75467a9fe59979ce8936cd. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:20.930804 7159 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5d18a2918c75467a9fe59979ce8936cd with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:20.931516 7167 catalog_manager.cc:5484] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 8 to 9, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:20.932572 9318 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
W20240501 08:05:21.001927 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 08:05:21.009590 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:05:21.025537 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:05:21.111202 14189 consensus_queue.cc:579] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Leader has been unable to successfully communicate with peer 3b69c5d799e14e438df760453438aa75 for more than 300 seconds (300.067s)
I20240501 08:05:21.114106 7709 consensus_queue.cc:237] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
W20240501 08:05:21.115720 7646 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): Couldn't send request to peer b4f8020266b5455b872a2973fc396578. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:21.115733 7446 raft_consensus.cc:1261] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:05:21.116159 14189 consensus_queue.cc:1035] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:05:21.116425 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:21.117870 14183 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
I20240501 08:05:21.119243 7162 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 07967f68a412411494c6dd1f47209009 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:21.119408 7446 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
I20240501 08:05:21.120378 7167 catalog_manager.cc:5484] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 6 to 7, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:21.154711 14187 consensus_queue.cc:579] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Leader has been unable to successfully communicate with peer 3b69c5d799e14e438df760453438aa75 for more than 300 seconds (300.172s)
I20240501 08:05:21.160830 7313 consensus_queue.cc:237] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:05:21.162362 9057 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:21.162492 7446 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:21.162756 14187 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:05:21.163358 14187 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:05:21.164819 13570 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:05:21.166419 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:21.166482 7446 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:05:21.167560 9057 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:05:21.169278 7167 catalog_manager.cc:5484] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 8 to 9, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:05:21.173202 7313 consensus_queue.cc:237] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:05:21.174721 9057 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:05:21.174890 7446 raft_consensus.cc:1261] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:05:21.175235 14187 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:05:21.175380 13570 consensus_queue.cc:1035] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:05:21.175716 7250 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d8ae3c82c9d4c5c858fa3f9d5959eb7. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:21.177003 14272 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:05:21.177632 9057 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d8aa5ac376a54e1f8417e5af11d28fa7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:05:21.178932 7162 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3d8ae3c82c9d4c5c858fa3f9d5959eb7 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:21.179531 7446 raft_consensus.cc:2932] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:05:21.179885 7175 catalog_manager.cc:5484] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 9 to 10, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:05:21.186268 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:21.238726 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:05:21.275002 13367 consensus_queue.cc:579] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Leader has been unable to successfully communicate with peer 3b69c5d799e14e438df760453438aa75 for more than 300 seconds (300.341s)
I20240501 08:05:21.281291 7575 consensus_queue.cc:237] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:05:21.282739 7446 raft_consensus.cc:1261] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 08:05:21.282899 7514 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4297f95243ac431099a8279c3d9e4b25. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:21.283179 14279 consensus_queue.cc:1035] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:05:21.283370 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:21.285030 14278 raft_consensus.cc:2932] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:05:21.285413 7446 raft_consensus.cc:2932] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:05:21.286743 7159 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4297f95243ac431099a8279c3d9e4b25 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:21.288010 7167 catalog_manager.cc:5484] T 4297f95243ac431099a8279c3d9e4b25 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 6 to 7, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:05:21.318454 14283 ts_tablet_manager.cc:902] T 81ccdd855b584d65860fd7a5777920a3 P d8aa5ac376a54e1f8417e5af11d28fa7: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
F20240501 08:05:21.318737 14283 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-02b42b5fd3d440ea950b35ed461b8e12.queue_time_us, tablet-download-pool-02b42b5fd3d440ea950b35ed461b8e12.queue_time_us) (tablet-download-pool-02b42b5fd3d440ea950b35ed461b8e12.run_cpu_time_us, tablet-download-pool-02b42b5fd3d440ea950b35ed461b8e12.run_cpu_time_us) (tablet-download-pool-02b42b5fd3d440ea950b35ed461b8e12.run_wall_time_us, tablet-download-pool-02b42b5fd3d440ea950b35ed461b8e12.run_wall_time_us) (tablet-download-pool-1e44704a9b9c46cf954d8e6937983841.queue_time_us, tablet-download-pool-1e44704a9b9c46cf954d8e6937983841.queue_time_us) (tablet-download-pool-1e44704a9b9c46cf954d8e6937983841.run_cpu_time_us, tablet-download-pool-1e44704a9b9c46cf954d8e6937983841.run_cpu_time_us) (tablet-download-pool-1e44704a9b9c46cf954d8e6937983841.run_wall_time_us, tablet-download-pool-1e44704a9b9c46cf954d8e6937983841.run_wall_time_us) (tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.queue_time_us, tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.queue_time_us) (tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.run_cpu_time_us, tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.run_cpu_time_us) (tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.run_wall_time_us, tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.run_wall_time_us) (tablet-download-pool-3d8ae3c82c9d4c5c858fa3f9d5959eb7.queue_time_us, tablet-download-pool-3d8ae3c82c9d4c5c858fa3f9d5959eb7.queue_time_us) (tablet-download-pool-3d8ae3c82c9d4c5c858fa3f9d5959eb7.run_cpu_time_us, tablet-download-pool-3d8ae3c82c9d4c5c858fa3f9d5959eb7.run_cpu_time_us) (tablet-download-pool-3d8ae3c82c9d4c5c858fa3f9d5959eb7.run_wall_time_us, tablet-download-pool-3d8ae3c82c9d4c5c858fa3f9d5959eb7.run_wall_time_us) (tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.queue_time_us, tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.queue_time_us) (tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.run_cpu_time_us, tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.run_cpu_time_us) (tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.run_wall_time_us, tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.run_wall_time_us) (tablet-download-pool-54ced80b9ff943ec88c27b7842500b3d.queue_time_us, tablet-download-pool-54ced80b9ff943ec88c27b7842500b3d.queue_time_us) (tablet-download-pool-54ced80b9ff943ec88c27b7842500b3d.run_cpu_time_us, tablet-download-pool-54ced80b9ff943ec88c27b7842500b3d.run_cpu_time_us) (tablet-download-pool-54ced80b9ff943ec88c27b7842500b3d.run_wall_time_us, tablet-download-pool-54ced80b9ff943ec88c27b7842500b3d.run_wall_time_us) (tablet-download-pool-5a37e631ae4f46a2be1ad5c43db5724b.queue_time_us, tablet-download-pool-5a37e631ae4f46a2be1ad5c43db5724b.queue_time_us) (tablet-download-pool-5a37e631ae4f46a2be1ad5c43db5724b.run_cpu_time_us, tablet-download-pool-5a37e631ae4f46a2be1ad5c43db5724b.run_cpu_time_us) (tablet-download-pool-5a37e631ae4f46a2be1ad5c43db5724b.run_wall_time_us, tablet-download-pool-5a37e631ae4f46a2be1ad5c43db5724b.run_wall_time_us) (tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.queue_time_us, tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.queue_time_us) (tablet-download-pool-8311d0921e9e417090e441884615525a.queue_time_us, tablet-download-pool-8311d0921e9e417090e441884615525a.queue_time_us) (tablet-download-pool-8311d0921e9e417090e441884615525a.run_cpu_time_us, tablet-download-pool-8311d0921e9e417090e441884615525a.run_cpu_time_us) (tablet-download-pool-8311d0921e9e417090e441884615525a.run_wall_time_us, tablet-download-pool-8311d0921e9e417090e441884615525a.run_wall_time_us) (tablet-download-pool-8882528b26974c7aa73bb0ff0c444836.queue_time_us, tablet-download-pool-8882528b26974c7aa73bb0ff0c444836.queue_time_us) (tablet-download-pool-8882528b26974c7aa73bb0ff0c444836.run_cpu_time_us, tablet-download-pool-8882528b26974c7aa73bb0ff0c444836.run_cpu_time_us) (tablet-download-pool-8882528b26974c7aa73bb0ff0c444836.run_wall_time_us, tablet-download-pool-8882528b26974c7aa73bb0ff0c444836.run_wall_time_us) (tablet-download-pool-b357c740aca144b7b063845726f419e2.queue_time_us, tablet-download-pool-b357c740aca144b7b063845726f419e2.queue_time_us) (tablet-download-pool-b357c740aca144b7b063845726f419e2.run_cpu_time_us, tablet-download-pool-b357c740aca144b7b063845726f419e2.run_cpu_time_us) (tablet-download-pool-b357c740aca144b7b063845726f419e2.run_wall_time_us, tablet-download-pool-b357c740aca144b7b063845726f419e2.run_wall_time_us) (tablet-download-pool-cc6f7dcc930e4937865d37cb3c3bb3d2.queue_time_us, tablet-download-pool-cc6f7dcc930e4937865d37cb3c3bb3d2.queue_time_us) (tablet-download-pool-cc6f7dcc930e4937865d37cb3c3bb3d2.run_cpu_time_us, tablet-download-pool-cc6f7dcc930e4937865d37cb3c3bb3d2.run_cpu_time_us) (tablet-download-pool-cc6f7dcc930e4937865d37cb3c3bb3d2.run_wall_time_us, tablet-download-pool-cc6f7dcc930e4937865d37cb3c3bb3d2.run_wall_time_us) (tablet-download-pool-df6dd3d8482d4931b1d3014a67e739c6.queue_time_us, tablet-download-pool-df6dd3d8482d4931b1d3014a67e739c6.queue_time_us) (tablet-download-pool-df6dd3d8482d4931b1d3014a67e739c6.run_cpu_time_us, tablet-download-pool-df6dd3d8482d4931b1d3014a67e739c6.run_cpu_time_us) (tablet-download-pool-df6dd3d8482d4931b1d3014a67e739c6.run_wall_time_us, tablet-download-pool-df6dd3d8482d4931b1d3014a67e739c6.run_wall_time_us) (tablet-download-pool-e178f7c3c17c4e1ca3d16b6441f76e77.queue_time_us, tablet-download-pool-e178f7c3c17c4e1ca3d16b6441f76e77.queue_time_us) (tablet-download-pool-e178f7c3c17c4e1ca3d16b6441f76e77.run_cpu_time_us, tablet-download-pool-e178f7c3c17c4e1ca3d16b6441f76e77.run_cpu_time_us) (tablet-download-pool-e178f7c3c17c4e1ca3d16b6441f76e77.run_wall_time_us, tablet-download-pool-e178f7c3c17c4e1ca3d16b6441f76e77.run_wall_time_us) (tablet-download-pool-ffc23756b44d4d588e0a8175075d9bc6.queue_time_us, tablet-download-pool-ffc23756b44d4d588e0a8175075d9bc6.queue_time_us) (tablet-download-pool-ffc23756b44d4d588e0a8175075d9bc6.run_cpu_time_us, tablet-download-pool-ffc23756b44d4d588e0a8175075d9bc6.run_cpu_time_us) (tablet-download-pool-ffc23756b44d4d588e0a8175075d9bc6.run_wall_time_us, tablet-download-pool-ffc23756b44d4d588e0a8175075d9bc6.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f4797a99dcd google::LogMessage::Fail() at ??:0
@ 0x7f4797a9db93 google::LogMessage::SendToLog() at ??:0
@ 0x7f4797a997cc google::LogMessage::Flush() at ??:0
@ 0x7f4797a9af59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4798f03044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f4798ee7f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f4798ee5efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f479ba9877a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f479baa1a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f479bb2276f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f479bb364f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f479bb21226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f479bb30c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5569d4dca75c std::function<>::operator()() at ??:0
@ 0x7f4798eeae06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4798eeb6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4798eed03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5569d4dca75c std::function<>::operator()() at ??:0
@ 0x7f4798eda807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f47997186db start_thread at ??:0
@ 0x7f4796c5071f clone at ??:0
I20240501 08:05:21.396282 14187 consensus_queue.cc:579] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Leader has been unable to successfully communicate with peer 1807c0c2e7d04b20b0cda311c42eef8e for more than 300 seconds (300.331s)
W20240501 08:05:21.398969 7251 consensus_peers.cc:462] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:05:21.399744 7316 consensus_queue.cc:237] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:05:21.401152 9319 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 08:05:21.401762 7446 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 08:05:21.402299 13570 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:05:21.402572 13570 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:05:21.405246 14187 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:05:21.407085 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1088a116ccec4020aec562a69791a188 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:21.408267 7446 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:05:21.408404 7171 catalog_manager.cc:5484] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 12 to 13, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:21.411159 9319 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:05:21.417336 7316 consensus_queue.cc:237] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:05:21.418632 9319 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 08:05:21.418673 7446 raft_consensus.cc:1261] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 08:05:21.419071 13570 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:05:21.419250 14187 consensus_queue.cc:1035] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:05:21.421350 14272 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:05:21.421967 7446 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:05:21.423029 9318 raft_consensus.cc:2932] T 1088a116ccec4020aec562a69791a188 P 1c8cb9b1059a47cc822ca13a2b931776 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:05:21.423533 7162 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1088a116ccec4020aec562a69791a188 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:21.424829 7175 catalog_manager.cc:5484] T 1088a116ccec4020aec562a69791a188 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 13 to 14, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:05:21.434791 14327 ts_tablet_manager.cc:902] T 5d18a2918c75467a9fe59979ce8936cd P b4f8020266b5455b872a2973fc396578: Initiating tablet copy from peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707)
I20240501 08:05:21.435431 14327 tablet_copy_client.cc:308] T 5d18a2918c75467a9fe59979ce8936cd P b4f8020266b5455b872a2973fc396578: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40707
I20240501 08:05:21.435992 7467 tablet_copy_service.cc:140] P d6b4be7bbd1649d0b7c5e8efe81f759f: Received BeginTabletCopySession request for tablet 5d18a2918c75467a9fe59979ce8936cd from peer b4f8020266b5455b872a2973fc396578 ({username='slave'} at 127.0.99.129:39821)
I20240501 08:05:21.436121 7467 tablet_copy_service.cc:161] P d6b4be7bbd1649d0b7c5e8efe81f759f: Beginning new tablet copy session on tablet 5d18a2918c75467a9fe59979ce8936cd from peer b4f8020266b5455b872a2973fc396578 at {username='slave'} at 127.0.99.129:39821: session id = b4f8020266b5455b872a2973fc396578-5d18a2918c75467a9fe59979ce8936cd
I20240501 08:05:21.437705 7467 tablet_copy_source_session.cc:205] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:05:21.439119 14327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d18a2918c75467a9fe59979ce8936cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:21.441673 14327 tablet_copy_client.cc:783] T 5d18a2918c75467a9fe59979ce8936cd P b4f8020266b5455b872a2973fc396578: tablet copy: Starting download of 5 data blocks...
I20240501 08:05:21.447652 14327 tablet_copy_client.cc:647] T 5d18a2918c75467a9fe59979ce8936cd P b4f8020266b5455b872a2973fc396578: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:21.449139 14327 tablet_copy_client.cc:523] T 5d18a2918c75467a9fe59979ce8936cd P b4f8020266b5455b872a2973fc396578: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:21.450928 14327 tablet_bootstrap.cc:492] T 5d18a2918c75467a9fe59979ce8936cd P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
W20240501 08:05:21.458969 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:05:21.461709 14327 tablet_bootstrap.cc:492] T 5d18a2918c75467a9fe59979ce8936cd P b4f8020266b5455b872a2973fc396578: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:21.462045 14327 tablet_bootstrap.cc:492] T 5d18a2918c75467a9fe59979ce8936cd P b4f8020266b5455b872a2973fc396578: Bootstrap complete.
I20240501 08:05:21.462247 14327 ts_tablet_manager.cc:1367] T 5d18a2918c75467a9fe59979ce8936cd P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.011s user 0.011s sys 0.000s
I20240501 08:05:21.462759 14327 raft_consensus.cc:348] T 5d18a2918c75467a9fe59979ce8936cd P b4f8020266b5455b872a2973fc396578 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
I20240501 08:05:21.462872 14327 raft_consensus.cc:729] T 5d18a2918c75467a9fe59979ce8936cd P b4f8020266b5455b872a2973fc396578 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: LEARNER
I20240501 08:05:21.463012 14327 consensus_queue.cc:260] T 5d18a2918c75467a9fe59979ce8936cd P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
I20240501 08:05:21.463297 14327 ts_tablet_manager.cc:1398] T 5d18a2918c75467a9fe59979ce8936cd P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:05:21.464251 7467 tablet_copy_service.cc:342] P d6b4be7bbd1649d0b7c5e8efe81f759f: Request end of tablet copy session b4f8020266b5455b872a2973fc396578-5d18a2918c75467a9fe59979ce8936cd received from {username='slave'} at 127.0.99.129:39821
I20240501 08:05:21.464339 7467 tablet_copy_service.cc:433] P d6b4be7bbd1649d0b7c5e8efe81f759f: ending tablet copy session b4f8020266b5455b872a2973fc396578-5d18a2918c75467a9fe59979ce8936cd on tablet 5d18a2918c75467a9fe59979ce8936cd with peer b4f8020266b5455b872a2973fc396578
I20240501 08:05:21.473892 14189 consensus_queue.cc:579] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Leader has been unable to successfully communicate with peer 3b69c5d799e14e438df760453438aa75 for more than 300 seconds (300.322s)
I20240501 08:05:21.476958 7709 consensus_queue.cc:237] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:05:21.478642 7446 raft_consensus.cc:1261] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:21.479140 14189 consensus_queue.cc:1035] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:05:21.479665 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:21.481431 14186 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:05:21.483038 7162 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4ea0db42a75a4879b3d3d608666dc2bf with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:21.483531 7446 raft_consensus.cc:2932] T 4ea0db42a75a4879b3d3d608666dc2bf P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:05:21.484158 7171 catalog_manager.cc:5484] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 8 to 9, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:05:21.492941 7644 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:05:21.533561 14279 consensus_queue.cc:579] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Leader has been unable to successfully communicate with peer 1807c0c2e7d04b20b0cda311c42eef8e for more than 300 seconds (300.060s)
I20240501 08:05:21.536792 7575 consensus_queue.cc:237] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } }
I20240501 08:05:21.538697 7313 raft_consensus.cc:1261] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:05:21.539202 14282 consensus_queue.cc:1035] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:05:21.539413 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:21.541173 14279 raft_consensus.cc:2932] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:05:21.541738 7313 raft_consensus.cc:2932] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:05:21.543380 7159 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cd07a00b6817400f86bf8c32bd592d8b with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:21.544701 7167 catalog_manager.cc:5484] T cd07a00b6817400f86bf8c32bd592d8b P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 6 to 7, NON_VOTER d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: true } } }
I20240501 08:05:21.562695 14189 consensus_queue.cc:579] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Leader has been unable to successfully communicate with peer 1807c0c2e7d04b20b0cda311c42eef8e for more than 300 seconds (300.211s)
W20240501 08:05:21.566054 7645 connection.cc:535] client connection to 127.0.99.133:33901 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:05:21.566059 7514 connection.cc:535] client connection to 127.0.99.133:33901 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:05:21.566285 7709 consensus_queue.cc:237] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:05:21.566463 7252 connection.cc:535] client connection to 127.0.99.133:33901 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:05:21.566584 7383 connection.cc:535] client connection to 127.0.99.133:33901 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:05:21.567298 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:21.567354 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.567394 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.567437 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.567601 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.568293 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.568367 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.568426 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.568495 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.568730 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.569229 7643 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffc23756b44d4d588e0a8175075d9bc6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:21.643234 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling UndoDeltaBlockGCOp(5d18a2918c75467a9fe59979ce8936cd): 4839 bytes on disk
W20240501 08:05:21.643674 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:21.643744 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: UndoDeltaBlockGCOp(5d18a2918c75467a9fe59979ce8936cd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240501 08:05:21.646296 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:21.680266 14327 ts_tablet_manager.cc:902] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:05:21.680747 14327 tablet_copy_client.cc:308] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:05:21.681216 7728 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 07967f68a412411494c6dd1f47209009 from peer b4f8020266b5455b872a2973fc396578 ({username='slave'} at 127.0.99.129:43719)
I20240501 08:05:21.681332 7728 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 07967f68a412411494c6dd1f47209009 from peer b4f8020266b5455b872a2973fc396578 at {username='slave'} at 127.0.99.129:43719: session id = b4f8020266b5455b872a2973fc396578-07967f68a412411494c6dd1f47209009
I20240501 08:05:21.682699 7728 tablet_copy_source_session.cc:205] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 5 blocks and 1 log segments
W20240501 08:05:21.683207 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:21.683575 14327 ts_tablet_manager.cc:1875] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:05:21.685897 14327 ts_tablet_manager.cc:1888] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 08:05:21.686115 14327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07967f68a412411494c6dd1f47209009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:21.687093 14327 tablet_copy_client.cc:783] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: tablet copy: Starting download of 5 data blocks...
I20240501 08:05:21.691673 14327 tablet_copy_client.cc:647] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:21.692229 14272 consensus_queue.cc:579] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Leader has been unable to successfully communicate with peer 1807c0c2e7d04b20b0cda311c42eef8e for more than 300 seconds (300.314s)
I20240501 08:05:21.693295 14327 tablet_copy_client.cc:523] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 08:05:21.694739 7251 consensus_peers.cc:462] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
I20240501 08:05:21.695111 7313 consensus_queue.cc:237] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } }
I20240501 08:05:21.695380 14327 tablet_bootstrap.cc:492] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:05:21.696213 14375 ts_tablet_manager.cc:902] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P 1c8cb9b1059a47cc822ca13a2b931776: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
F20240501 08:05:21.696422 14375 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0e42ef8df4884bc9bc33f5ed4c164c8c.queue_time_us, tablet-download-pool-0e42ef8df4884bc9bc33f5ed4c164c8c.queue_time_us) (tablet-download-pool-0e42ef8df4884bc9bc33f5ed4c164c8c.run_cpu_time_us, tablet-download-pool-0e42ef8df4884bc9bc33f5ed4c164c8c.run_cpu_time_us) (tablet-download-pool-0e42ef8df4884bc9bc33f5ed4c164c8c.run_wall_time_us, tablet-download-pool-0e42ef8df4884bc9bc33f5ed4c164c8c.run_wall_time_us) (tablet-download-pool-1088a116ccec4020aec562a69791a188.queue_time_us, tablet-download-pool-1088a116ccec4020aec562a69791a188.queue_time_us) (tablet-download-pool-1088a116ccec4020aec562a69791a188.run_cpu_time_us, tablet-download-pool-1088a116ccec4020aec562a69791a188.run_cpu_time_us) (tablet-download-pool-1088a116ccec4020aec562a69791a188.run_wall_time_us, tablet-download-pool-1088a116ccec4020aec562a69791a188.run_wall_time_us) (tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.queue_time_us, tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.queue_time_us) (tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.run_cpu_time_us, tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.run_cpu_time_us) (tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.run_wall_time_us, tablet-download-pool-21d76484f5dd499fb6fc3d534bffb779.run_wall_time_us) (tablet-download-pool-28baea6d7d3f442ba1dbfc5b37de90ac.queue_time_us, tablet-download-pool-28baea6d7d3f442ba1dbfc5b37de90ac.queue_time_us) (tablet-download-pool-28baea6d7d3f442ba1dbfc5b37de90ac.run_cpu_time_us, tablet-download-pool-28baea6d7d3f442ba1dbfc5b37de90ac.run_cpu_time_us) (tablet-download-pool-28baea6d7d3f442ba1dbfc5b37de90ac.run_wall_time_us, tablet-download-pool-28baea6d7d3f442ba1dbfc5b37de90ac.run_wall_time_us) (tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.queue_time_us, tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.queue_time_us) (tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.run_cpu_time_us, tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.run_cpu_time_us) (tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.run_wall_time_us, tablet-download-pool-3abb5202d8d24d9b9a67f7a87b7300a6.run_wall_time_us) (tablet-download-pool-3d8ae3c82c9d4c5c858fa3f9d5959eb7.queue_time_us, tablet-download-pool-3d8ae3c82c9d4c5c858fa3f9d5959eb7.queue_time_us) (tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.queue_time_us, tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.queue_time_us) (tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.run_cpu_time_us, tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.run_cpu_time_us) (tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.run_wall_time_us, tablet-download-pool-4dde9710a5eb4baa881d9dd04affe184.run_wall_time_us) (tablet-download-pool-5970772c02d04e0b95a2f2feb7dcd655.queue_time_us, tablet-download-pool-5970772c02d04e0b95a2f2feb7dcd655.queue_time_us) (tablet-download-pool-5970772c02d04e0b95a2f2feb7dcd655.run_cpu_time_us, tablet-download-pool-5970772c02d04e0b95a2f2feb7dcd655.run_cpu_time_us) (tablet-download-pool-5970772c02d04e0b95a2f2feb7dcd655.run_wall_time_us, tablet-download-pool-5970772c02d04e0b95a2f2feb7dcd655.run_wall_time_us) (tablet-download-pool-5d18a2918c75467a9fe59979ce8936cd.queue_time_us, tablet-download-pool-5d18a2918c75467a9fe59979ce8936cd.queue_time_us) (tablet-download-pool-5d18a2918c75467a9fe59979ce8936cd.run_cpu_time_us, tablet-download-pool-5d18a2918c75467a9fe59979ce8936cd.run_cpu_time_us) (tablet-download-pool-5d18a2918c75467a9fe59979ce8936cd.run_wall_time_us, tablet-download-pool-5d18a2918c75467a9fe59979ce8936cd.run_wall_time_us) (tablet-download-pool-7377810e08104526b8b8acdf886a3368.queue_time_us, tablet-download-pool-7377810e08104526b8b8acdf886a3368.queue_time_us) (tablet-download-pool-7377810e08104526b8b8acdf886a3368.run_cpu_time_us, tablet-download-pool-7377810e08104526b8b8acdf886a3368.run_cpu_time_us) (tablet-download-pool-7377810e08104526b8b8acdf886a3368.run_wall_time_us, tablet-download-pool-7377810e08104526b8b8acdf886a3368.run_wall_time_us) (tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.queue_time_us, tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.queue_time_us) (tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.run_cpu_time_us, tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.run_cpu_time_us) (tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.run_wall_time_us, tablet-download-pool-81ccdd855b584d65860fd7a5777920a3.run_wall_time_us) (tablet-download-pool-8311d0921e9e417090e441884615525a.queue_time_us, tablet-download-pool-8311d0921e9e417090e441884615525a.queue_time_us) (tablet-download-pool-8311d0921e9e417090e441884615525a.run_cpu_time_us, tablet-download-pool-8311d0921e9e417090e441884615525a.run_cpu_time_us) (tablet-download-pool-8311d0921e9e417090e441884615525a.run_wall_time_us, tablet-download-pool-8311d0921e9e417090e441884615525a.run_wall_time_us) (tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.queue_time_us, tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.queue_time_us) (tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.run_cpu_time_us, tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.run_cpu_time_us) (tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.run_wall_time_us, tablet-download-pool-bde42e485c4b49ce8acd694a25ba683a.run_wall_time_us) (tablet-download-pool-df6dd3d8482d4931b1d3014a67e739c6.queue_time_us, tablet-download-pool-df6dd3d8482d4931b1d3014a67e739c6.queue_time_us) (tablet-download-pool-df6dd3d8482d4931b1d3014a67e739c6.run_cpu_time_us, tablet-download-pool-df6dd3d8482d4931b1d3014a67e739c6.run_cpu_time_us) (tablet-download-pool-df6dd3d8482d4931b1d3014a67e739c6.run_wall_time_us, tablet-download-pool-df6dd3d8482d4931b1d3014a67e739c6.run_wall_time_us) (tablet-download-pool-e178f7c3c17c4e1ca3d16b6441f76e77.queue_time_us, tablet-download-pool-e178f7c3c17c4e1ca3d16b6441f76e77.queue_time_us) (tablet-download-pool-e178f7c3c17c4e1ca3d16b6441f76e77.run_cpu_time_us, tablet-download-pool-e178f7c3c17c4e1ca3d16b6441f76e77.run_cpu_time_us) (tablet-download-pool-e178f7c3c17c4e1ca3d16b6441f76e77.run_wall_time_us, tablet-download-pool-e178f7c3c17c4e1ca3d16b6441f76e77.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
W20240501 08:05:21.697257 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.697402 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f55aaf7bdcd google::LogMessage::Fail() at ??:0
@ 0x7f55aaf7fb93 google::LogMessage::SendToLog() at ??:0
I20240501 08:05:21.704138 14327 tablet_bootstrap.cc:492] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
@ 0x7f55aaf7b7cc google::LogMessage::Flush() at ??:0
I20240501 08:05:21.704382 14327 tablet_bootstrap.cc:492] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: Bootstrap complete.
I20240501 08:05:21.704512 14327 ts_tablet_manager.cc:1367] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240501 08:05:21.704854 14327 raft_consensus.cc:348] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
I20240501 08:05:21.704947 14327 raft_consensus.cc:729] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: LEARNER
I20240501 08:05:21.705019 14327 consensus_queue.cc:260] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
I20240501 08:05:21.705264 14327 ts_tablet_manager.cc:1398] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:05:21.705801 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session b4f8020266b5455b872a2973fc396578-07967f68a412411494c6dd1f47209009 received from {username='slave'} at 127.0.99.129:43719
I20240501 08:05:21.705902 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session b4f8020266b5455b872a2973fc396578-07967f68a412411494c6dd1f47209009 on tablet 07967f68a412411494c6dd1f47209009 with peer b4f8020266b5455b872a2973fc396578
@ 0x7f55aaf7cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f55ac3e5044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f55ac3c9f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f55ac3c7efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f55aef7a77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f55aef83a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f55af00476f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f55af0184f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f55af003226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f55af012c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55bc53e2175c std::function<>::operator()() at ??:0
@ 0x7f55ac3cce06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f55ac3cd6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f55ac3cf03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55bc53e2175c std::function<>::operator()() at ??:0
@ 0x7f55ac3bc807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f55acbfa6db start_thread at ??:0
@ 0x7f55aa13271f clone at ??:0
W20240501 08:05:21.756115 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.782888 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:21.809012 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:21.812580 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.819252 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.819334 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.840821 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.850138 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:21.877571 7313 raft_consensus.cc:1203] T 5d18a2918c75467a9fe59979ce8936cd P b4f8020266b5455b872a2973fc396578 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 08:05:21.895274 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling UndoDeltaBlockGCOp(07967f68a412411494c6dd1f47209009): 4586 bytes on disk
I20240501 08:05:21.895736 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: UndoDeltaBlockGCOp(07967f68a412411494c6dd1f47209009) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240501 08:05:21.900240 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.926039 7512 connection.cc:535] client connection to 127.0.99.135:41471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:05:21.926061 7643 connection.cc:535] client connection to 127.0.99.135:41471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:05:21.926084 7250 connection.cc:535] client connection to 127.0.99.135:41471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:05:21.926390 7381 connection.cc:535] client connection to 127.0.99.135:41471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:05:21.927076 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:21.927157 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.927273 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.927402 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.927472 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.927524 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.927527 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:21.927608 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:21.940112 13367 consensus_queue.cc:579] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Leader has been unable to successfully communicate with peer 3b69c5d799e14e438df760453438aa75 for more than 300 seconds (300.096s)
W20240501 08:05:21.941399 7513 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:21.942868 7575 consensus_queue.cc:237] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:05:21.944380 7709 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240501 08:05:21.944748 13367 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240501 08:05:21.944875 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:21.946477 14282 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:05:21.946966 7709 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:05:21.948552 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21d76484f5dd499fb6fc3d534bffb779 with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:21.949196 7171 catalog_manager.cc:5484] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 15 to 16, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:21.952327 7575 consensus_queue.cc:237] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } }
W20240501 08:05:21.952399 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 21d76484f5dd499fb6fc3d534bffb779: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:05:21.953631 7709 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
W20240501 08:05:21.953970 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707): Couldn't send request to peer d6b4be7bbd1649d0b7c5e8efe81f759f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:21.954154 14282 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20240501 08:05:21.954576 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:21.955861 13367 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, NON_VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } } }
I20240501 08:05:21.956251 7709 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, NON_VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } } }
I20240501 08:05:21.957177 7159 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 21d76484f5dd499fb6fc3d534bffb779 with cas_config_opid_index 16: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:21.958526 7171 catalog_manager.cc:5484] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 16 to 17, NON_VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:05:21.960686 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:21.974155 14186 consensus_queue.cc:579] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Leader has been unable to successfully communicate with peer 1807c0c2e7d04b20b0cda311c42eef8e for more than 300 seconds (300.137s)
W20240501 08:05:22.004638 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:22.023751 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:22.031811 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:22.046793 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:22.075511 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
I20240501 08:05:22.092571 14215 consensus_queue.cc:579] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Leader has been unable to successfully communicate with peer 3b69c5d799e14e438df760453438aa75 for more than 300 seconds (300.449s)
I20240501 08:05:22.095424 7446 consensus_queue.cc:237] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } }
I20240501 08:05:22.096787 7575 raft_consensus.cc:1261] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:22.097191 14215 consensus_queue.cc:1035] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:05:22.097358 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:22.099228 14398 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:05:22.099668 7575 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } }
I20240501 08:05:22.100795 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f647cfc562794e65a05918b241014d72 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:22.102114 7167 catalog_manager.cc:5484] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 8 to 9, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:22.105188 7446 consensus_queue.cc:237] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:05:22.106657 7575 raft_consensus.cc:1261] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 08:05:22.107049 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:22.107090 14397 consensus_queue.cc:1035] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:05:22.107223 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:22.108422 14215 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:05:22.108820 7575 raft_consensus.cc:2932] T f647cfc562794e65a05918b241014d72 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:05:22.110404 7159 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f647cfc562794e65a05918b241014d72 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:22.110785 7171 catalog_manager.cc:5484] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 9 to 10, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:22.116752 13570 consensus_queue.cc:579] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Leader has been unable to successfully communicate with peer 1807c0c2e7d04b20b0cda311c42eef8e for more than 300 seconds (300.409s)
W20240501 08:05:22.118347 7251 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:22.120494 7313 consensus_queue.cc:237] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:05:22.121945 7575 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:22.122391 13570 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:05:22.122803 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:22.124259 14272 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:05:22.124658 7575 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:05:22.126232 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b357c740aca144b7b063845726f419e2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:22.127120 7167 catalog_manager.cc:5484] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 8 to 9, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:22.130916 7313 consensus_queue.cc:237] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } }
I20240501 08:05:22.132277 7575 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 08:05:22.132513 7250 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707): Couldn't send request to peer d6b4be7bbd1649d0b7c5e8efe81f759f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b357c740aca144b7b063845726f419e2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:22.133527 14187 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:05:22.133934 7513 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240501 08:05:22.134936 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:22.135457 13570 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } } }
I20240501 08:05:22.136080 7575 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } } }
I20240501 08:05:22.137101 7162 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b357c740aca144b7b063845726f419e2 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:22.138126 7171 catalog_manager.cc:5484] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 9 to 10, NON_VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:05:22.167296 7644 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 08:05:22.173880 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 08:05:22.189530 7313 raft_consensus.cc:1203] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240501 08:05:22.195295 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:22.198385 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
I20240501 08:05:22.348098 13367 consensus_queue.cc:579] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Leader has been unable to successfully communicate with peer 1807c0c2e7d04b20b0cda311c42eef8e for more than 300 seconds (300.228s)
I20240501 08:05:22.381686 14452 ts_tablet_manager.cc:902] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:05:22.382329 14452 tablet_copy_client.cc:308] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:05:22.382966 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet 21d76484f5dd499fb6fc3d534bffb779 from peer d6b4be7bbd1649d0b7c5e8efe81f759f ({username='slave'} at 127.0.99.130:50251)
I20240501 08:05:22.383087 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet 21d76484f5dd499fb6fc3d534bffb779 from peer d6b4be7bbd1649d0b7c5e8efe81f759f at {username='slave'} at 127.0.99.130:50251: session id = d6b4be7bbd1649d0b7c5e8efe81f759f-21d76484f5dd499fb6fc3d534bffb779
I20240501 08:05:22.384573 7599 tablet_copy_source_session.cc:205] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:05:22.385653 14452 ts_tablet_manager.cc:1875] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:05:22.387629 14452 ts_tablet_manager.cc:1888] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240501 08:05:22.387810 14452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d76484f5dd499fb6fc3d534bffb779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:22.388860 14452 tablet_copy_client.cc:783] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet copy: Starting download of 5 data blocks...
I20240501 08:05:22.393730 14452 tablet_copy_client.cc:647] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:22.394965 14452 tablet_copy_client.cc:523] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:22.396761 14452 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
W20240501 08:05:22.397338 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
I20240501 08:05:22.408375 14452 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:22.408670 14452 tablet_bootstrap.cc:492] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap complete.
I20240501 08:05:22.408843 14452 ts_tablet_manager.cc:1367] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.001s
I20240501 08:05:22.409257 14452 raft_consensus.cc:348] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } }
I20240501 08:05:22.409364 14452 raft_consensus.cc:729] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: LEARNER
I20240501 08:05:22.409451 14452 consensus_queue.cc:260] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 1.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } }
I20240501 08:05:22.409730 14452 ts_tablet_manager.cc:1398] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:05:22.410286 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session d6b4be7bbd1649d0b7c5e8efe81f759f-21d76484f5dd499fb6fc3d534bffb779 received from {username='slave'} at 127.0.99.130:50251
I20240501 08:05:22.410363 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session d6b4be7bbd1649d0b7c5e8efe81f759f-21d76484f5dd499fb6fc3d534bffb779 on tablet 21d76484f5dd499fb6fc3d534bffb779 with peer d6b4be7bbd1649d0b7c5e8efe81f759f
W20240501 08:05:22.417189 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 47 similar messages]
I20240501 08:05:22.428148 14397 raft_consensus.cc:1050] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f: attempting to promote NON_VOTER b4f8020266b5455b872a2973fc396578 to VOTER
I20240501 08:05:22.428637 14397 consensus_queue.cc:237] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } }
I20240501 08:05:22.430087 7313 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P b4f8020266b5455b872a2973fc396578 [term 1 LEARNER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:05:22.430413 7575 raft_consensus.cc:1261] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:05:22.430771 14398 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:05:22.430953 14215 consensus_queue.cc:1035] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:05:22.431088 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:22.438197 14398 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:22.439435 7313 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:22.440369 7575 raft_consensus.cc:2932] T 5d18a2918c75467a9fe59979ce8936cd P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:22.441537 7167 catalog_manager.cc:5484] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 9 to 10, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:05:22.457195 13367 consensus_queue.cc:579] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Leader has been unable to successfully communicate with peer 1807c0c2e7d04b20b0cda311c42eef8e for more than 300 seconds (300.233s)
W20240501 08:05:22.458606 7513 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:05:22.459901 7575 consensus_queue.cc:237] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } }
I20240501 08:05:22.461308 7313 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
W20240501 08:05:22.461658 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:22.461756 13367 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 08:05:22.463515 13367 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:22.463914 7313 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:22.465225 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bde42e485c4b49ce8acd694a25ba683a with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:22.466310 7175 catalog_manager.cc:5484] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 14 to 15, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:22.469558 7575 consensus_queue.cc:237] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } }
I20240501 08:05:22.470762 7313 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
W20240501 08:05:22.471000 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707): Couldn't send request to peer d6b4be7bbd1649d0b7c5e8efe81f759f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:22.471110 14279 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240501 08:05:22.472322 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:22.472700 14282 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } } }
I20240501 08:05:22.473140 7313 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } } }
I20240501 08:05:22.474514 7159 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bde42e485c4b49ce8acd694a25ba683a with cas_config_opid_index 15: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:22.475453 7171 catalog_manager.cc:5484] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 15 to 16, NON_VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:22.521623 14189 consensus_queue.cc:579] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Leader has been unable to successfully communicate with peer 3b69c5d799e14e438df760453438aa75 for more than 300 seconds (300.503s)
I20240501 08:05:22.522508 14452 ts_tablet_manager.cc:902] T b357c740aca144b7b063845726f419e2 P d6b4be7bbd1649d0b7c5e8efe81f759f: Initiating tablet copy from peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263)
I20240501 08:05:22.522995 14452 tablet_copy_client.cc:308] T b357c740aca144b7b063845726f419e2 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:40263
I20240501 08:05:22.523485 7336 tablet_copy_service.cc:140] P b4f8020266b5455b872a2973fc396578: Received BeginTabletCopySession request for tablet b357c740aca144b7b063845726f419e2 from peer d6b4be7bbd1649d0b7c5e8efe81f759f ({username='slave'} at 127.0.99.130:55793)
I20240501 08:05:22.523602 7336 tablet_copy_service.cc:161] P b4f8020266b5455b872a2973fc396578: Beginning new tablet copy session on tablet b357c740aca144b7b063845726f419e2 from peer d6b4be7bbd1649d0b7c5e8efe81f759f at {username='slave'} at 127.0.99.130:55793: session id = d6b4be7bbd1649d0b7c5e8efe81f759f-b357c740aca144b7b063845726f419e2
I20240501 08:05:22.524696 7709 consensus_queue.cc:237] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }
I20240501 08:05:22.524900 7336 tablet_copy_source_session.cc:205] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:05:22.526026 14452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b357c740aca144b7b063845726f419e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:22.526204 7575 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:22.526794 13214 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:05:22.527002 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:22.528445 14452 tablet_copy_client.cc:783] T b357c740aca144b7b063845726f419e2 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet copy: Starting download of 5 data blocks...
I20240501 08:05:22.528972 13214 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:05:22.529644 7575 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } }
I20240501 08:05:22.531391 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5a37e631ae4f46a2be1ad5c43db5724b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:22.531971 7171 catalog_manager.cc:5484] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 8 to 9, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:22.533975 14452 tablet_copy_client.cc:647] T b357c740aca144b7b063845726f419e2 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:22.535303 14452 tablet_copy_client.cc:523] T b357c740aca144b7b063845726f419e2 P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:22.535866 7709 consensus_queue.cc:237] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } }
I20240501 08:05:22.536530 14452 tablet_bootstrap.cc:492] T b357c740aca144b7b063845726f419e2 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
W20240501 08:05:22.537233 7643 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130:40707): Couldn't send request to peer d6b4be7bbd1649d0b7c5e8efe81f759f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:22.537636 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:22.537685 7575 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:05:22.538028 14189 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:05:22.539393 14186 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } } }
I20240501 08:05:22.539837 7575 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } } }
I20240501 08:05:22.541566 7162 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5a37e631ae4f46a2be1ad5c43db5724b with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:22.542088 7175 catalog_manager.cc:5484] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 9 to 10, NON_VOTER d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:22.546933 14452 tablet_bootstrap.cc:492] T b357c740aca144b7b063845726f419e2 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:22.547196 14452 tablet_bootstrap.cc:492] T b357c740aca144b7b063845726f419e2 P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap complete.
I20240501 08:05:22.547351 14452 ts_tablet_manager.cc:1367] T b357c740aca144b7b063845726f419e2 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240501 08:05:22.547773 14452 raft_consensus.cc:348] T b357c740aca144b7b063845726f419e2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } }
I20240501 08:05:22.547906 14452 raft_consensus.cc:729] T b357c740aca144b7b063845726f419e2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: LEARNER
I20240501 08:05:22.548020 14452 consensus_queue.cc:260] T b357c740aca144b7b063845726f419e2 P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } }
I20240501 08:05:22.548310 14452 ts_tablet_manager.cc:1398] T b357c740aca144b7b063845726f419e2 P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:05:22.548959 7336 tablet_copy_service.cc:342] P b4f8020266b5455b872a2973fc396578: Request end of tablet copy session d6b4be7bbd1649d0b7c5e8efe81f759f-b357c740aca144b7b063845726f419e2 received from {username='slave'} at 127.0.99.130:55793
I20240501 08:05:22.549054 7336 tablet_copy_service.cc:433] P b4f8020266b5455b872a2973fc396578: ending tablet copy session d6b4be7bbd1649d0b7c5e8efe81f759f-b357c740aca144b7b063845726f419e2 on tablet b357c740aca144b7b063845726f419e2 with peer d6b4be7bbd1649d0b7c5e8efe81f759f
W20240501 08:05:22.551947 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
I20240501 08:05:22.570700 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling UndoDeltaBlockGCOp(b357c740aca144b7b063845726f419e2): 4839 bytes on disk
I20240501 08:05:22.571239 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: UndoDeltaBlockGCOp(b357c740aca144b7b063845726f419e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240501 08:05:22.571966 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling UndoDeltaBlockGCOp(21d76484f5dd499fb6fc3d534bffb779): 4586 bytes on disk
I20240501 08:05:22.572353 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: UndoDeltaBlockGCOp(21d76484f5dd499fb6fc3d534bffb779) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240501 08:05:22.604466 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:22.698052 7382 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:22.767419 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
I20240501 08:05:22.789017 13214 raft_consensus.cc:1050] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER b4f8020266b5455b872a2973fc396578 to VOTER
I20240501 08:05:22.789451 13214 consensus_queue.cc:237] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } }
I20240501 08:05:22.790844 7313 raft_consensus.cc:1261] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:05:22.790851 7446 raft_consensus.cc:1261] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:05:22.791229 14499 consensus_queue.cc:1035] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:05:22.791364 13214 consensus_queue.cc:1035] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:05:22.791805 7644 consensus_peers.cc:485] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:22.793576 13214 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:22.794060 7446 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:22.796437 7171 catalog_manager.cc:5484] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 7 to 8, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:22.797259 7313 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b69c5d799e14e438df760453438aa75" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33589 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:22.802207 7709 consensus_queue.cc:237] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } }
I20240501 08:05:22.803481 7446 raft_consensus.cc:1261] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:22.803817 7313 raft_consensus.cc:1261] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:22.803872 14186 consensus_queue.cc:1035] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:05:22.804147 13214 consensus_queue.cc:1035] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:05:22.805534 14189 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:22.807031 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 07967f68a412411494c6dd1f47209009 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:22.807432 7446 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:22.807432 7313 raft_consensus.cc:2932] T 07967f68a412411494c6dd1f47209009 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:22.810108 7171 catalog_manager.cc:5484] T 07967f68a412411494c6dd1f47209009 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 8 to 9, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:05:22.895308 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
I20240501 08:05:22.899551 7446 raft_consensus.cc:1203] T b357c740aca144b7b063845726f419e2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 08:05:22.921947 14452 ts_tablet_manager.cc:902] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:05:22.922513 14452 tablet_copy_client.cc:308] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:05:22.923013 7728 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 5a37e631ae4f46a2be1ad5c43db5724b from peer d6b4be7bbd1649d0b7c5e8efe81f759f ({username='slave'} at 127.0.99.130:59719)
I20240501 08:05:22.923120 7728 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 5a37e631ae4f46a2be1ad5c43db5724b from peer d6b4be7bbd1649d0b7c5e8efe81f759f at {username='slave'} at 127.0.99.130:59719: session id = d6b4be7bbd1649d0b7c5e8efe81f759f-5a37e631ae4f46a2be1ad5c43db5724b
I20240501 08:05:22.924166 7728 tablet_copy_source_session.cc:205] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:05:22.924916 14452 ts_tablet_manager.cc:1875] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:05:22.926915 14452 ts_tablet_manager.cc:1888] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 08:05:22.927084 14452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a37e631ae4f46a2be1ad5c43db5724b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:22.927969 14452 tablet_copy_client.cc:783] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet copy: Starting download of 5 data blocks...
I20240501 08:05:22.932570 14452 tablet_copy_client.cc:647] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:22.933681 14452 tablet_copy_client.cc:523] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:22.934846 14452 tablet_bootstrap.cc:492] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:05:22.943470 14452 tablet_bootstrap.cc:492] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:22.943781 14452 tablet_bootstrap.cc:492] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap complete.
I20240501 08:05:22.943976 14452 ts_tablet_manager.cc:1367] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240501 08:05:22.944411 14452 raft_consensus.cc:348] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } }
I20240501 08:05:22.944523 14452 raft_consensus.cc:729] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: LEARNER
I20240501 08:05:22.944623 14452 consensus_queue.cc:260] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } }
I20240501 08:05:22.944913 14452 ts_tablet_manager.cc:1398] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:05:22.945500 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session d6b4be7bbd1649d0b7c5e8efe81f759f-5a37e631ae4f46a2be1ad5c43db5724b received from {username='slave'} at 127.0.99.130:59719
I20240501 08:05:22.945658 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session d6b4be7bbd1649d0b7c5e8efe81f759f-5a37e631ae4f46a2be1ad5c43db5724b on tablet 5a37e631ae4f46a2be1ad5c43db5724b with peer d6b4be7bbd1649d0b7c5e8efe81f759f
I20240501 08:05:22.978402 7446 raft_consensus.cc:1203] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.16->[1.17-1.17] Dedup: 1.17->[]
W20240501 08:05:22.996701 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 07967f68a412411494c6dd1f47209009: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
I20240501 08:05:23.028867 14452 ts_tablet_manager.cc:902] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: Initiating tablet copy from peer 31b02f06be8a4455b94b1254d77005bd (127.0.99.131:37861)
I20240501 08:05:23.029327 14452 tablet_copy_client.cc:308] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37861
I20240501 08:05:23.029831 7599 tablet_copy_service.cc:140] P 31b02f06be8a4455b94b1254d77005bd: Received BeginTabletCopySession request for tablet bde42e485c4b49ce8acd694a25ba683a from peer d6b4be7bbd1649d0b7c5e8efe81f759f ({username='slave'} at 127.0.99.130:50251)
I20240501 08:05:23.029948 7599 tablet_copy_service.cc:161] P 31b02f06be8a4455b94b1254d77005bd: Beginning new tablet copy session on tablet bde42e485c4b49ce8acd694a25ba683a from peer d6b4be7bbd1649d0b7c5e8efe81f759f at {username='slave'} at 127.0.99.130:50251: session id = d6b4be7bbd1649d0b7c5e8efe81f759f-bde42e485c4b49ce8acd694a25ba683a
I20240501 08:05:23.030788 7599 tablet_copy_source_session.cc:205] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:05:23.031464 14452 ts_tablet_manager.cc:1875] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:05:23.033232 14452 ts_tablet_manager.cc:1888] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240501 08:05:23.033412 14452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde42e485c4b49ce8acd694a25ba683a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:23.034289 14452 tablet_copy_client.cc:783] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet copy: Starting download of 5 data blocks...
I20240501 08:05:23.039659 14452 tablet_copy_client.cc:647] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:23.041054 14452 tablet_copy_client.cc:523] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:23.042487 14452 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap starting.
I20240501 08:05:23.053637 14452 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:23.053957 14452 tablet_bootstrap.cc:492] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: Bootstrap complete.
I20240501 08:05:23.054167 14452 ts_tablet_manager.cc:1367] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.002s
I20240501 08:05:23.054623 14452 raft_consensus.cc:348] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } }
I20240501 08:05:23.054777 14452 raft_consensus.cc:729] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d6b4be7bbd1649d0b7c5e8efe81f759f, State: Initialized, Role: LEARNER
I20240501 08:05:23.054886 14452 consensus_queue.cc:260] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: true } }
I20240501 08:05:23.055153 14452 ts_tablet_manager.cc:1398] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:05:23.055879 7599 tablet_copy_service.cc:342] P 31b02f06be8a4455b94b1254d77005bd: Request end of tablet copy session d6b4be7bbd1649d0b7c5e8efe81f759f-bde42e485c4b49ce8acd694a25ba683a received from {username='slave'} at 127.0.99.130:50251
I20240501 08:05:23.055972 7599 tablet_copy_service.cc:433] P 31b02f06be8a4455b94b1254d77005bd: ending tablet copy session d6b4be7bbd1649d0b7c5e8efe81f759f-bde42e485c4b49ce8acd694a25ba683a on tablet bde42e485c4b49ce8acd694a25ba683a with peer d6b4be7bbd1649d0b7c5e8efe81f759f
I20240501 08:05:23.074926 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling UndoDeltaBlockGCOp(5a37e631ae4f46a2be1ad5c43db5724b): 4839 bytes on disk
I20240501 08:05:23.075331 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: UndoDeltaBlockGCOp(5a37e631ae4f46a2be1ad5c43db5724b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240501 08:05:23.075958 7495 maintenance_manager.cc:419] P d6b4be7bbd1649d0b7c5e8efe81f759f: Scheduling UndoDeltaBlockGCOp(bde42e485c4b49ce8acd694a25ba683a): 4837 bytes on disk
I20240501 08:05:23.076246 7385 maintenance_manager.cc:643] P d6b4be7bbd1649d0b7c5e8efe81f759f: UndoDeltaBlockGCOp(bde42e485c4b49ce8acd694a25ba683a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4}
W20240501 08:05:23.084517 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:23.127172 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:23.130537 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
I20240501 08:05:23.131304 14513 consensus_queue.cc:579] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Leader has been unable to successfully communicate with peer 1807c0c2e7d04b20b0cda311c42eef8e for more than 300 seconds (300.229s)
I20240501 08:05:23.134370 7575 consensus_queue.cc:237] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } }
I20240501 08:05:23.135959 7709 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:05:23.136369 14512 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 08:05:23.136807 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:23.138135 14513 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
I20240501 08:05:23.138620 7709 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
I20240501 08:05:23.139925 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cc6f7dcc930e4937865d37cb3c3bb3d2 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:23.141816 7171 catalog_manager.cc:5484] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 11 to 12, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } }
W20240501 08:05:23.142302 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:23.144954 7513 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:23.145273 7575 consensus_queue.cc:237] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:05:23.146560 7709 raft_consensus.cc:1261] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 08:05:23.146994 14512 consensus_queue.cc:1035] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 08:05:23.147132 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:23.147202 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:23.148466 13367 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:05:23.148913 7709 raft_consensus.cc:2932] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:05:23.149933 7159 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cc6f7dcc930e4937865d37cb3c3bb3d2 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:23.151350 7175 catalog_manager.cc:5484] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 12 to 13, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:05:23.182390 14186 consensus_queue.cc:579] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [LEADER]: Leader has been unable to successfully communicate with peer 3b69c5d799e14e438df760453438aa75 for more than 300 seconds (300.452s)
I20240501 08:05:23.185194 7709 consensus_queue.cc:237] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } }
I20240501 08:05:23.186650 7575 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 2 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20240501 08:05:23.186861 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:23.187067 14189 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:05:23.188717 13214 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:05:23.189108 7575 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } }
I20240501 08:05:23.190224 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 28baea6d7d3f442ba1dbfc5b37de90ac with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:23.191514 7171 catalog_manager.cc:5484] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 8 to 10, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New cstate: current_term: 2 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:23.198644 7709 consensus_queue.cc:237] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
I20240501 08:05:23.200223 7575 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 2 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
W20240501 08:05:23.200407 7646 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer b4f8020266b5455b872a2973fc396578 (127.0.99.129:40263): Couldn't send request to peer b4f8020266b5455b872a2973fc396578. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:23.200755 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:23.200874 14189 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:05:23.202734 14186 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
I20240501 08:05:23.203215 7575 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } } }
I20240501 08:05:23.204480 7162 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 28baea6d7d3f442ba1dbfc5b37de90ac with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:23.205675 7171 catalog_manager.cc:5484] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 10 to 11, NON_VOTER b4f8020266b5455b872a2973fc396578 (127.0.99.129) added. New cstate: current_term: 2 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:05:23.281220 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
I20240501 08:05:23.295660 14215 consensus_queue.cc:579] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Leader has been unable to successfully communicate with peer 1807c0c2e7d04b20b0cda311c42eef8e for more than 300 seconds (300.023s)
I20240501 08:05:23.298413 7446 consensus_queue.cc:237] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } }
I20240501 08:05:23.299635 7709 raft_consensus.cc:1261] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:23.299672 7575 raft_consensus.cc:1261] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:23.300071 14215 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:05:23.300210 14556 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:05:23.302088 14556 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:05:23.302515 7575 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:05:23.302599 7709 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:05:23.304003 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dd17760cd5c2454db97610b2f71c0ee2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:23.305343 7175 catalog_manager.cc:5484] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 8 to 9, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:05:23.309072 7446 consensus_queue.cc:237] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:05:23.310269 7575 raft_consensus.cc:1261] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:05:23.310546 7709 raft_consensus.cc:1261] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer d6b4be7bbd1649d0b7c5e8efe81f759f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:05:23.310654 14215 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:05:23.310915 14555 consensus_queue.cc:1035] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:05:23.311725 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:23.312435 14215 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:05:23.312850 7709 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:05:23.313213 7575 raft_consensus.cc:2932] T dd17760cd5c2454db97610b2f71c0ee2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:05:23.314051 7159 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dd17760cd5c2454db97610b2f71c0ee2 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:23.315660 7167 catalog_manager.cc:5484] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 9 to 10, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:23.327682 7446 raft_consensus.cc:1203] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
W20240501 08:05:23.334702 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
I20240501 08:05:23.429204 13570 raft_consensus.cc:1050] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578: attempting to promote NON_VOTER d6b4be7bbd1649d0b7c5e8efe81f759f to VOTER
I20240501 08:05:23.429683 13570 consensus_queue.cc:237] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } }
I20240501 08:05:23.431075 7446 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEARNER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:05:23.431172 7575 raft_consensus.cc:1261] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer b4f8020266b5455b872a2973fc396578: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:05:23.431507 14563 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:05:23.431811 14562 consensus_queue.cc:1035] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 08:05:23.432021 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:23.440045 14563 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } } }
I20240501 08:05:23.440105 7575 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } } }
I20240501 08:05:23.442967 7446 raft_consensus.cc:2932] T b357c740aca144b7b063845726f419e2 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } } }
I20240501 08:05:23.443727 7175 catalog_manager.cc:5484] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 10 to 11, d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4f8020266b5455b872a2973fc396578" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:05:23.512717 7446 raft_consensus.cc:1203] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.15->[1.16-1.16] Dedup: 1.16->[]
W20240501 08:05:23.518981 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
I20240501 08:05:23.520730 14499 consensus_queue.cc:579] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Leader has been unable to successfully communicate with peer 1807c0c2e7d04b20b0cda311c42eef8e for more than 300 seconds (300.394s)
W20240501 08:05:23.521967 7644 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:05:23.523218 7709 consensus_queue.cc:237] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } }
I20240501 08:05:23.524502 7313 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 08:05:23.524960 14186 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240501 08:05:23.525111 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:23.526830 14499 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:23.527212 7313 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:23.528144 7162 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4dde9710a5eb4baa881d9dd04affe184 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:23.529610 7171 catalog_manager.cc:5484] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 14 to 15, VOTER 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:05:23.533135 7709 consensus_queue.cc:237] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:05:23.534440 7313 raft_consensus.cc:1261] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240501 08:05:23.534914 14189 consensus_queue.cc:1035] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240501 08:05:23.535177 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:23.535254 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:23.536458 14189 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:05:23.536828 7313 raft_consensus.cc:2932] T 4dde9710a5eb4baa881d9dd04affe184 P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:05:23.537725 7162 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4dde9710a5eb4baa881d9dd04affe184 with cas_config_opid_index 15: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:23.539225 7171 catalog_manager.cc:5484] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 15 to 16, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:23.558476 14512 raft_consensus.cc:1050] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER d6b4be7bbd1649d0b7c5e8efe81f759f to VOTER
I20240501 08:05:23.558898 14512 consensus_queue.cc:237] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } }
I20240501 08:05:23.560248 7446 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20240501 08:05:23.560308 7709 raft_consensus.cc:1261] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20240501 08:05:23.560616 13367 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240501 08:05:23.560757 14513 consensus_queue.cc:1035] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20240501 08:05:23.561401 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:23.562882 13367 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } } }
I20240501 08:05:23.563328 7709 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P 09a7b6e2731a474b80411efb62323604 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } } }
I20240501 08:05:23.565022 7446 raft_consensus.cc:2932] T 21d76484f5dd499fb6fc3d534bffb779 P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } } }
I20240501 08:05:23.565804 7167 catalog_manager.cc:5484] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 17 to 18, d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:05:23.578891 14586 ts_tablet_manager.cc:902] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: Initiating tablet copy from peer 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789)
I20240501 08:05:23.579751 14586 tablet_copy_client.cc:308] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:35789
I20240501 08:05:23.580408 7728 tablet_copy_service.cc:140] P 09a7b6e2731a474b80411efb62323604: Received BeginTabletCopySession request for tablet 28baea6d7d3f442ba1dbfc5b37de90ac from peer b4f8020266b5455b872a2973fc396578 ({username='slave'} at 127.0.99.129:43719)
I20240501 08:05:23.580499 7728 tablet_copy_service.cc:161] P 09a7b6e2731a474b80411efb62323604: Beginning new tablet copy session on tablet 28baea6d7d3f442ba1dbfc5b37de90ac from peer b4f8020266b5455b872a2973fc396578 at {username='slave'} at 127.0.99.129:43719: session id = b4f8020266b5455b872a2973fc396578-28baea6d7d3f442ba1dbfc5b37de90ac
I20240501 08:05:23.581380 7728 tablet_copy_source_session.cc:205] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:05:23.582090 14586 ts_tablet_manager.cc:1875] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:05:23.583930 14586 ts_tablet_manager.cc:1888] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 08:05:23.584071 14586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28baea6d7d3f442ba1dbfc5b37de90ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:23.584976 14586 tablet_copy_client.cc:783] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: tablet copy: Starting download of 5 data blocks...
I20240501 08:05:23.589219 14586 tablet_copy_client.cc:647] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:23.590415 14586 tablet_copy_client.cc:523] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:23.591682 14586 tablet_bootstrap.cc:492] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: Bootstrap starting.
I20240501 08:05:23.601011 14586 tablet_bootstrap.cc:492] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:23.601256 14586 tablet_bootstrap.cc:492] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: Bootstrap complete.
I20240501 08:05:23.601420 14586 ts_tablet_manager.cc:1367] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240501 08:05:23.601820 14586 raft_consensus.cc:348] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
I20240501 08:05:23.601931 14586 raft_consensus.cc:729] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b4f8020266b5455b872a2973fc396578, State: Initialized, Role: LEARNER
I20240501 08:05:23.602053 14586 consensus_queue.cc:260] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: true } }
I20240501 08:05:23.602355 14586 ts_tablet_manager.cc:1398] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 08:05:23.602823 7728 tablet_copy_service.cc:342] P 09a7b6e2731a474b80411efb62323604: Request end of tablet copy session b4f8020266b5455b872a2973fc396578-28baea6d7d3f442ba1dbfc5b37de90ac received from {username='slave'} at 127.0.99.129:43719
I20240501 08:05:23.602924 7728 tablet_copy_service.cc:433] P 09a7b6e2731a474b80411efb62323604: ending tablet copy session b4f8020266b5455b872a2973fc396578-28baea6d7d3f442ba1dbfc5b37de90ac on tablet 28baea6d7d3f442ba1dbfc5b37de90ac with peer b4f8020266b5455b872a2973fc396578
I20240501 08:05:23.651530 7364 maintenance_manager.cc:419] P b4f8020266b5455b872a2973fc396578: Scheduling UndoDeltaBlockGCOp(28baea6d7d3f442ba1dbfc5b37de90ac): 4926 bytes on disk
I20240501 08:05:23.651969 7254 maintenance_manager.cc:643] P b4f8020266b5455b872a2973fc396578: UndoDeltaBlockGCOp(28baea6d7d3f442ba1dbfc5b37de90ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240501 08:05:23.752868 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:23.786139 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:23.795653 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:23.816845 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 08:05:23.847407 14189 raft_consensus.cc:1050] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER d6b4be7bbd1649d0b7c5e8efe81f759f to VOTER
I20240501 08:05:23.847877 14189 consensus_queue.cc:237] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } }
I20240501 08:05:23.849361 7446 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:05:23.849462 7575 raft_consensus.cc:1261] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:05:23.849898 13214 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:05:23.850060 14186 consensus_queue.cc:1035] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 08:05:23.850358 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:23.852006 14186 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } } }
I20240501 08:05:23.852517 7446 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } } }
I20240501 08:05:23.853788 7575 raft_consensus.cc:2932] T 5a37e631ae4f46a2be1ad5c43db5724b P 31b02f06be8a4455b94b1254d77005bd [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } } }
I20240501 08:05:23.854619 7175 catalog_manager.cc:5484] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 10 to 11, d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:05:23.859004 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:23.868438 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:23.956527 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:23.963320 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:23.964174 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:05:23.965026 7313 raft_consensus.cc:1203] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
W20240501 08:05:24.012382 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.061003 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.064487 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:05:24.085695 7164 maintenance_manager.cc:419] P 6aa238de40d94e5f9898ac424a4d1f68: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 11545 bytes on disk
I20240501 08:05:24.087976 7163 maintenance_manager.cc:643] P 6aa238de40d94e5f9898ac424a4d1f68: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.002s sys 0.000s Metrics: {"dirs.queue_time_us":168,"thread_start_us":69,"threads_started":1}
I20240501 08:05:24.117884 14642 raft_consensus.cc:1050] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd: attempting to promote NON_VOTER d6b4be7bbd1649d0b7c5e8efe81f759f to VOTER
I20240501 08:05:24.118333 14642 consensus_queue.cc:237] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } }
I20240501 08:05:24.119858 7446 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 LEARNER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20240501 08:05:24.119853 7313 raft_consensus.cc:1261] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20240501 08:05:24.120268 13367 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240501 08:05:24.120396 14645 consensus_queue.cc:1035] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20240501 08:05:24.120496 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:24.122442 13367 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } } }
I20240501 08:05:24.122821 7446 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } } }
I20240501 08:05:24.124608 7313 raft_consensus.cc:2932] T bde42e485c4b49ce8acd694a25ba683a P b4f8020266b5455b872a2973fc396578 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } } }
I20240501 08:05:24.125432 7171 catalog_manager.cc:5484] T bde42e485c4b49ce8acd694a25ba683a P d6b4be7bbd1649d0b7c5e8efe81f759f reported cstate change: config changed from index 16 to 17, d6b4be7bbd1649d0b7c5e8efe81f759f (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } peers { permanent_uuid: "d6b4be7bbd1649d0b7c5e8efe81f759f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } attrs { promote: false } } }
W20240501 08:05:24.166039 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.167433 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.167953 7160 catalog_manager.cc:4573] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4dde9710a5eb4baa881d9dd04affe184: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:24.177440 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.203418 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.206013 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:05:24.232937 13367 consensus_queue.cc:579] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Leader has been unable to successfully communicate with peer 3b69c5d799e14e438df760453438aa75 for more than 300 seconds (300.346s)
I20240501 08:05:24.235844 7575 consensus_queue.cc:237] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } }
I20240501 08:05:24.237104 7313 raft_consensus.cc:1261] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 2 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:05:24.237104 7709 raft_consensus.cc:1261] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 2 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:05:24.237519 14642 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:05:24.237805 14642 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:05:24.237939 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:05:24.239555 14644 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
W20240501 08:05:24.240216 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:05:24.240618 7313 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:05:24.241380 7159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:24.241513 7709 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:05:24.243208 7175 catalog_manager.cc:5484] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 reported cstate change: config changed from index 8 to 10, VOTER 3b69c5d799e14e438df760453438aa75 (127.0.99.134) evicted. New cstate: current_term: 2 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } }
I20240501 08:05:24.246737 7575 consensus_queue.cc:237] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
I20240501 08:05:24.248116 7313 raft_consensus.cc:1261] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 2 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 08:05:24.248164 7709 raft_consensus.cc:1261] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 2 FOLLOWER]: Refusing update from remote peer 31b02f06be8a4455b94b1254d77005bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 08:05:24.248534 14644 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 08:05:24.248739 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:24.248888 13367 consensus_queue.cc:1035] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:05:24.250104 14644 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:05:24.250973 7709 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 09a7b6e2731a474b80411efb62323604 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:05:24.251634 7159 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240501 08:05:24.251781 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
I20240501 08:05:24.251513 7313 raft_consensus.cc:2932] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P b4f8020266b5455b872a2973fc396578 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } } }
I20240501 08:05:24.253464 7171 catalog_manager.cc:5484] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd reported cstate change: config changed from index 10 to 11, NON_VOTER 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135) added. New cstate: current_term: 2 leader_uuid: "31b02f06be8a4455b94b1254d77005bd" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:05:24.284994 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.292624 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.298938 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.304481 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.321635 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:24.346885 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.351259 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:24.354903 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:05:24.422138 13214 raft_consensus.cc:1050] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604: attempting to promote NON_VOTER b4f8020266b5455b872a2973fc396578 to VOTER
I20240501 08:05:24.422504 13214 consensus_queue.cc:237] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } }
I20240501 08:05:24.423766 7575 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 2 FOLLOWER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 08:05:24.423753 7313 raft_consensus.cc:1261] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 2 LEARNER]: Refusing update from remote peer 09a7b6e2731a474b80411efb62323604: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 08:05:24.424131 14667 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:05:24.424314 14186 consensus_queue.cc:1035] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 08:05:24.424440 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:24.426510 14186 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:24.427042 7575 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P 31b02f06be8a4455b94b1254d77005bd [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:24.428324 7313 raft_consensus.cc:2932] T 28baea6d7d3f442ba1dbfc5b37de90ac P b4f8020266b5455b872a2973fc396578 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } } }
I20240501 08:05:24.430289 7175 catalog_manager.cc:5484] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 reported cstate change: config changed from index 11 to 12, b4f8020266b5455b872a2973fc396578 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "09a7b6e2731a474b80411efb62323604" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b02f06be8a4455b94b1254d77005bd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f8020266b5455b872a2973fc396578" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40263 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:05:24.448408 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:24.491724 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.491832 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.512895 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.526847 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.546594 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.591289 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.700970 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:24.771529 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.798992 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:24.821261 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:24.866554 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 155 similar messages]
W20240501 08:05:24.870067 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:24.957654 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:24.977455 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:25.214582 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5d18a2918c75467a9fe59979ce8936cd: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:25.263065 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:25.284934 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:25.319523 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:25.548650 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:25.567507 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:25.579604 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 08:05:25.620540 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:25.753760 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:25.775445 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:25.827848 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:25.866772 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:25.911551 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:25.925094 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:25.930822 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:26.022930 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240501 08:05:26.079223 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:26.213910 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:26.269948 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240501 08:05:26.270618 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 21d76484f5dd499fb6fc3d534bffb779: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:26.320737 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:26.339543 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:26.366986 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:26.434566 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:26.450847 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:26.456872 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.465370 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 08:05:26.478331 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.483325 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:26.490162 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:26.507478 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.531188 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:26.545367 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.572408 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.589867 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.593084 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.597609 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.633329 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.657156 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.657730 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.686878 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.704164 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.735791 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.754226 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:26.762583 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:26.777282 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.783067 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.783102 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.824039 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.862133 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.874873 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.889565 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.910017 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:26.928112 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.970777 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:26.974258 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:27.055442 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:27.062216 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:27.142925 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:27.146188 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:27.159595 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:27.203763 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:27.314224 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240501 08:05:27.346343 7160 catalog_manager.cc:4573] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cc6f7dcc930e4937865d37cb3c3bb3d2: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:27.372413 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:27.379946 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:27.461366 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:27.474437 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:05:27.562106 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:27.583779 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:27.683238 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:27.879983 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:27.968860 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:27.997025 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:28.078896 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:28.082458 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:28.131464 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:28.204862 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:28.225955 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240501 08:05:28.266907 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:28.345947 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:28.368570 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:28.400115 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:28.435086 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:28.492240 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:28.571816 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:28.585104 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:28.656306 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:28.675341 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:28.763938 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:28.821568 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:28.855432 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:28.855629 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:28.879575 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:28.943814 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:28.964771 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:28.983911 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 08:05:29.045725 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:29.075619 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.103713 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:29.109838 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.119397 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.121111 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.154316 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.190941 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.236451 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:29.236959 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:29.245077 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.251433 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.254642 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.270633 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.287092 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.307574 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.348893 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:29.364552 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.393244 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.418907 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.444679 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:29.451506 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.458153 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.507599 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.511984 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.526422 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.593664 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:29.598847 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.608204 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.608870 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.654938 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:29.683707 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:29.756947 7160 catalog_manager.cc:4573] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1088a116ccec4020aec562a69791a188: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:29.803810 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:29.932742 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 160 similar messages]
W20240501 08:05:29.933419 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:30.000620 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:30.101253 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:30.141774 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:30.192305 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:30.229163 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:30.383586 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:30.435317 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:30.442309 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:30.460325 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:30.654856 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:30.679986 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:30.689540 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:30.710080 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:30.725287 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:30.895746 7160 catalog_manager.cc:4573] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bde42e485c4b49ce8acd694a25ba683a: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:30.972333 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:30.980787 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:30.986526 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 08:05:30.988394 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:31.084183 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:31.118722 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.143875 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 08:05:31.182166 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.191380 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:31.200516 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:31.261334 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:31.300637 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.329756 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:31.345366 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.361362 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 08:05:31.467247 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 08:05:31.506564 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.550285 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.555284 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:31.556499 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:31.584100 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:31.600610 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:31.637328 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.667678 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.697773 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.731372 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.748243 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.784617 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.801414 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.826491 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.838743 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.877027 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.911756 7160 catalog_manager.cc:4573] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4dde9710a5eb4baa881d9dd04affe184: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:31.965178 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.969885 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:31.975258 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:31.978178 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:31.989063 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:32.006948 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:32.017692 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:32.018167 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:32.023975 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:32.039727 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:32.049356 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:32.068591 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:32.085553 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:32.092130 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:32.144965 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:32.205747 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:32.214129 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:32.378240 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:32.404309 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240501 08:05:32.404973 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:32.477214 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:05:32.488096 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:32.582834 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:32.719261 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:32.763764 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:32.834313 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:32.878939 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:32.910655 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:32.988335 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:33.016621 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:33.043843 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:33.118052 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:33.171545 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:33.277069 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:33.294680 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:33.409251 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:33.447224 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:33.461762 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:33.463165 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:33.478551 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:33.551770 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:33.672515 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 08:05:33.695130 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:33.707280 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:33.721946 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:33.735757 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:33.791357 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:33.802675 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:33.866578 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:33.866590 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 08:05:33.930281 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:33.959957 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:34.058008 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.059836 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:34.091388 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.124035 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.124713 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.221149 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.243418 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:34.267197 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.269006 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.280660 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:34.281035 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.346004 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.386520 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.398439 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.399377 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.401046 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.430698 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:34.452688 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.462239 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:34.482939 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:34.486774 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:34.581960 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.653788 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:34.659193 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.664011 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.673344 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.673724 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.733443 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.766849 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.843796 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:34.934923 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240501 08:05:34.949596 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:35.036264 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:35.063684 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:35.107257 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:35.115643 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:35.189028 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:35.221443 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:35.415994 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:35.461899 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:35.482646 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:35.544914 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:35.561019 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:35.659765 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:35.739650 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:35.830626 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:35.836812 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:35.928128 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:35.947765 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:35.953461 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:35.957917 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:36.037626 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:36.067787 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:36.075752 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:36.124540 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:36.124691 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:36.179669 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:36.191841 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 08:05:36.216742 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 08:05:36.230978 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:36.262480 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:36.349566 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:36.372392 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:36.374589 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 08:05:36.383929 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:36.384058 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:36.488857 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:36.515585 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:36.539528 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:36.550627 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:36.568617 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:36.590525 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:36.629086 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:36.729266 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:36.809671 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:36.861423 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:36.886332 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:36.890164 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:36.940930 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:36.950866 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:36.999823 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:37.018515 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:37.060350 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:37.087566 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:37.098788 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:37.099419 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:37.150506 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:37.167835 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:37.184165 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:37.218294 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:37.249504 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:37.294862 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:37.333636 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:37.355873 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:37.386730 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:37.417927 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:37.423846 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:37.428411 7250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 08:05:37.479358 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:05:37.504220 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:37.572441 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240501 08:05:37.573137 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5d18a2918c75467a9fe59979ce8936cd: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:37.577613 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:37.629107 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:37.767515 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:37.798019 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:37.925653 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:37.992574 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:38.008121 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:38.029495 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:38.121153 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:38.154174 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:38.245153 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:38.252370 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:38.267395 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:38.291491 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:38.318614 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:38.388458 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:38.429013 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 08:05:38.442095 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:38.470652 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:38.475296 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:38.553053 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:38.600379 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:38.617821 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 21d76484f5dd499fb6fc3d534bffb779: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:38.649539 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:38.656631 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:38.674636 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:38.701887 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:38.703366 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:38.725823 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:38.832821 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 08:05:38.939474 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:38.940598 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:38.986876 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.010419 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.062657 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:39.122331 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.140846 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:39.176173 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:39.194936 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.273399 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.319051 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.334581 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:39.369431 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:39.429510 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.444311 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:39.528442 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.538574 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:39.550350 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.563222 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.589370 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.621114 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.633432 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.647130 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.657385 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:39.679574 7160 catalog_manager.cc:4573] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cc6f7dcc930e4937865d37cb3c3bb3d2: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:39.704689 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:39.719696 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.732237 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.755463 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.780843 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.783674 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.812350 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:39.849831 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.875931 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:39.899713 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:39.968837 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 163 similar messages]
W20240501 08:05:39.971729 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:40.105967 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:40.140188 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:40.228046 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:40.296036 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:40.388098 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:40.417109 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:40.582901 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:40.602936 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:40.610911 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:40.659777 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:40.724488 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:40.834707 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:40.845645 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:40.882169 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:40.886528 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:40.888106 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:40.927347 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:40.959118 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 08:05:41.001868 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:41.010486 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:41.022119 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:41.037482 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:41.067878 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:41.076143 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:41.112505 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:41.195010 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240501 08:05:41.204291 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:41.263103 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:41.266889 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:41.302521 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:41.345805 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 08:05:41.397485 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:41.404904 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:41.405329 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:41.443548 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:41.536211 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:41.577056 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:41.634090 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:41.688007 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:41.728320 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:41.754645 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:41.772657 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:41.852617 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:41.923920 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:41.927404 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:41.962756 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:42.045970 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:42.046587 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:42.097149 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:42.138547 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:42.208129 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:42.208619 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:42.252717 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:42.262161 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:42.283953 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:42.313652 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:42.327517 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:42.363205 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:42.380707 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:42.404280 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:42.425066 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:42.433257 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240501 08:05:42.446379 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:42.502664 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:42.507513 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:42.511885 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240501 08:05:42.512432 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:42.524904 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:42.808884 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:42.833804 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:42.890619 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:42.948696 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:42.972296 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:43.020623 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:43.179817 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:43.187320 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:43.209964 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:43.225450 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:43.307598 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:43.340871 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:43.369875 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:43.372031 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 08:05:43.391846 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:43.407791 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:43.431608 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:43.432884 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:43.533293 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:43.592025 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:43.740808 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:43.775074 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:43.813294 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:43.861577 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:43.869365 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 08:05:43.886301 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:43.899950 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:43.919139 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:43.924873 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:43.926717 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:43.955379 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:43.989791 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:44.047932 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:44.063375 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:44.077679 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:44.179457 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:44.194790 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:44.221216 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:44.340090 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:44.355052 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:44.359143 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:44.404371 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:44.437887 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:44.469683 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:44.517808 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:44.617130 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:44.684329 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:44.684899 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:44.748744 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:44.792449 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:44.825192 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:44.856575 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:44.874472 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:44.925591 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:44.926982 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:44.950104 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:44.956568 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:44.975637 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240501 08:05:44.980953 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:45.016238 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:45.035264 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:45.035903 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:45.057479 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:45.067010 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:45.360057 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:45.427945 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:45.474829 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:45.509303 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:45.545629 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:45.560397 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:45.609932 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:45.614862 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:45.646560 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:45.814517 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 08:05:45.845516 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:45.946839 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:45.950644 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:45.971277 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:45.985224 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:46.002175 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:46.026504 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:46.040822 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:46.059513 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:46.156072 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:46.166572 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:46.246728 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:46.252040 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 08:05:46.252622 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:46.312749 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:46.319996 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:46.325505 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 08:05:46.434278 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:46.458493 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:46.461567 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:46.514086 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:46.516126 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:46.568238 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:46.575899 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:46.576077 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:46.612915 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:46.627013 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:46.636741 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:46.686547 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:46.745501 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:46.830499 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:46.891641 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:46.928059 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:46.957006 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:47.036723 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:47.037104 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:47.079778 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:47.126459 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:47.146513 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:47.161999 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:47.169847 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:47.203752 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:47.252580 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:47.282176 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:47.328883 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:47.334398 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:47.353422 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:47.359457 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:47.399309 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:47.442684 7250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 08:05:47.451859 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:47.461253 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:47.498256 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:47.553190 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:05:47.602968 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:47.621351 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:47.626148 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:47.704176 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:47.928311 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:47.951066 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:47.959908 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:48.064231 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:48.097342 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:48.125216 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:48.208236 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:48.238234 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 08:05:48.245060 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:48.283041 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:48.423942 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:48.497730 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:48.497706 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:48.526185 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:48.534466 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:48.552371 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:48.613045 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:48.658267 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:48.700076 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:48.767426 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:48.771968 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:48.821563 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:48.892527 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:48.908300 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:48.910163 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:48.925412 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:48.981537 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.000811 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 08:05:49.016785 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.048481 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:49.059278 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.088238 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:49.143097 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:49.191437 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.237275 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:49.254174 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.273413 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.292621 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.300834 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:49.354457 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:49.467799 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.526513 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:49.547268 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.552386 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:49.611289 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.672683 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.672751 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.695441 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.740295 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.744289 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:49.746557 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.783201 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:49.808640 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.838224 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.861055 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.903548 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:49.905931 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.939280 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:49.974792 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:50.001021 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 159 similar messages]
W20240501 08:05:50.001845 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:50.061286 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:50.143679 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:50.195505 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:50.254802 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:50.281311 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:50.290356 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:50.426118 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:50.499557 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:50.531829 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:50.548558 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:50.565810 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:50.711323 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:50.780076 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:50.832396 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:50.847048 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 08:05:50.869477 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:50.890558 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:51.064273 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:51.070809 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:51.077273 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:51.085667 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:51.104348 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:51.139667 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:51.173957 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:51.215468 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:51.269012 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:51.293613 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 08:05:51.332314 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:51.337242 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:51.460270 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:51.471333 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 08:05:51.524878 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:51.533133 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:51.557880 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:51.565882 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:51.566695 7162 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: ChangeConfig RPC to 127.0.99.132:35789 timed out after 30.000s (SENT) [suppressed 13 similar messages]
W20240501 08:05:51.568085 7162 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ffc23756b44d4d588e0a8175075d9bc6 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:05:51.569615 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:51.582937 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:51.591053 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:51.658711 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:51.717145 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:51.750773 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:51.764559 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:51.766230 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:51.799366 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:51.899364 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:51.914697 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:51.943948 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:51.990041 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:52.026845 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:52.129017 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:52.135421 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:52.185639 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:52.186388 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:52.206873 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:52.287863 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:52.314349 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:52.321007 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:52.368595 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:52.420858 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:52.438292 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:52.456738 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:52.492246 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:52.524098 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240501 08:05:52.538962 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:52.558238 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:52.599654 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:52.634984 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:52.652781 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:05:52.695152 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:52.715622 7162 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ffc23756b44d4d588e0a8175075d9bc6 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:05:52.746569 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:52.748020 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:52.816082 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:52.830047 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:53.066165 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:53.082686 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:53.190680 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:53.215370 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:53.339133 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:53.347682 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:53.351974 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:53.367420 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:53.408727 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:53.494541 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 08:05:53.508278 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:53.531952 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:53.550535 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:53.570281 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:53.570513 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:53.594595 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:53.638550 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:53.666905 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:53.717586 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:53.738829 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:53.748662 7162 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ffc23756b44d4d588e0a8175075d9bc6 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:05:53.802887 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:53.809203 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:53.880515 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:53.881090 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:53.964308 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5d18a2918c75467a9fe59979ce8936cd: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:53.984038 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 08:05:53.992352 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:54.009619 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:54.021145 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:54.122386 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:54.164404 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:54.189752 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:54.195629 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:54.196110 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:54.201251 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:54.223412 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:54.321339 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:54.411021 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:54.443044 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:54.543000 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:54.579859 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:54.596655 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:54.612493 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:54.645814 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:54.647162 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:54.741607 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:54.826948 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:54.853016 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:54.901078 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:54.913082 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:54.993407 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:55.014494 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 159 similar messages]
W20240501 08:05:55.015255 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:55.018208 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 21d76484f5dd499fb6fc3d534bffb779: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:55.046696 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:55.065620 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:55.066715 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:55.072520 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:55.077242 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:55.081235 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:55.093659 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:55.158461 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:55.175200 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:55.191610 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:55.514176 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:55.516571 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:55.529152 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:55.542232 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:55.547232 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:55.549479 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:55.684734 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:55.714044 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:55.819866 7162 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ffc23756b44d4d588e0a8175075d9bc6 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:05:55.856204 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:55.877125 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:55.952507 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:55.966961 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:55.982096 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:56.012079 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 08:05:56.047041 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:56.068820 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:56.069491 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:56.099979 7160 catalog_manager.cc:4573] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cc6f7dcc930e4937865d37cb3c3bb3d2: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:05:56.101828 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:56.106758 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:56.119943 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:56.184227 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:56.194108 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:56.242363 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:56.250643 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:56.261549 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:56.298765 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240501 08:05:56.400182 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:56.434293 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:56.464453 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 08:05:56.559486 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:56.560495 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:56.609479 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:56.623194 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:56.631276 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:56.661657 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:56.661763 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:56.713523 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:56.746654 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:56.802378 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:56.884660 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:56.974771 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:57.028506 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:57.096493 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:57.156952 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:57.162966 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:57.187142 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:57.211808 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:57.224517 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:57.291564 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:57.310374 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20240501 08:05:57.311074 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:05:57.361150 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:57.416604 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:57.437615 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:57.486469 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:57.545871 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:57.561462 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:57.614967 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240501 08:05:57.616803 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:57.624562 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:57.632490 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:57.654978 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:57.664270 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:05:57.664973 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:57.681442 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:57.706385 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:57.706821 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:57.751598 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:57.812175 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:57.935595 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:57.988517 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:58.102356 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:58.189019 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:58.207548 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:58.248499 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:58.252519 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 08:05:58.289710 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:58.296495 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:58.297538 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:58.315477 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:58.395224 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:58.425139 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:58.545722 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:58.602042 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:58.642618 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:58.655409 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:58.655606 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:58.672856 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:58.792486 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:58.820834 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:58.842693 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:58.842712 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:58.846387 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:58.873845 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:58.912132 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:58.936508 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:58.939719 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 08:05:58.955096 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:59.027292 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:59.117497 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:59.162356 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:59.188968 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:59.262276 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:59.263760 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:59.297087 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:59.391494 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:59.401286 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:59.410744 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:59.466586 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:59.494314 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:59.496891 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:59.552345 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:59.587361 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:59.715349 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:59.725585 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:05:59.790479 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:05:59.840572 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:59.853520 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:59.928618 7162 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ffc23756b44d4d588e0a8175075d9bc6 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:06:00.007597 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:00.015041 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 160 similar messages]
W20240501 08:06:00.020251 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:00.050956 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:06:00.056116 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:00.098174 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:00.110445 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:00.113552 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:00.144562 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:00.230923 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:00.320868 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:00.321616 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:06:00.334681 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:00.374274 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:00.423758 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:00.505864 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:00.589246 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:00.621354 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 08:06:00.631088 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:00.649941 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:00.676066 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:00.765030 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:00.803263 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:00.807356 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:00.854277 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:00.870694 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:00.907071 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:01.006795 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:01.008800 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:01.049965 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:01.084270 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:01.148811 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:01.186466 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:01.200009 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:01.303179 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:01.317826 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:01.341001 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240501 08:06:01.346664 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:01.349524 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:01.419237 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:01.456678 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:01.475318 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:01.531961 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:01.547597 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:01.564014 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:01.619271 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:01.619875 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:01.635959 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 08:06:01.644114 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:01.666648 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:01.745671 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:01.763115 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:01.797502 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:01.864462 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:01.875479 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:01.884296 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:01.909420 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:01.960330 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:02.053740 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:02.110066 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:02.167515 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:02.171522 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:02.183656 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:02.241729 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:02.306495 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:02.389931 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:02.412111 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:02.495992 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:02.511209 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:02.554190 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:02.571717 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:02.613808 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:02.614086 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:02.666946 7250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240501 08:06:02.667351 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:02.685699 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:02.728912 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:02.857358 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:02.862443 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240501 08:06:02.871047 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:02.896560 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:02.913713 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:02.957373 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:06:02.993675 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:03.073787 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 08:06:03.082214 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:03.116737 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:03.192953 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:03.195075 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:03.223140 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:03.252992 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:03.283010 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:03.304136 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:03.314090 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:03.350798 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:03.370929 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:03.418251 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:03.515280 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:03.590790 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:03.649547 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:03.704542 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:03.754738 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:03.784008 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:03.789036 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:03.797475 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:03.810377 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:03.829093 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:03.866568 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:03.954298 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:04.009014 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:04.021442 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:04.087697 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:04.132601 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 08:06:04.222124 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:04.222790 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:04.251716 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:04.263857 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:04.273392 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:04.275487 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:04.315862 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:04.359411 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:04.424319 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:04.460266 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:04.554577 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:04.561650 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:04.653868 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:04.655189 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:04.699052 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:04.742691 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:04.802294 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:04.810928 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:04.830194 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:04.980542 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:05.035948 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:05.055516 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 159 similar messages]
W20240501 08:06:05.123427 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:05.128369 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:05.139147 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:05.195480 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:05.293056 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:05.293401 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:05.342342 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:05.358618 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:05.398963 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:05.399982 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:05.443372 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:05.490990 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:05.504411 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:06:05.597247 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:05.644084 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:05.678599 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:05.684489 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:05.739388 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:05.770746 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:05.789733 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:05.790431 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 08:06:05.792855 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:05.887305 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:05.929275 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:05.959337 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:05.968478 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:05.992492 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:06.019713 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:06.041177 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:06.118536 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:06.212356 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:06.227563 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:06.249104 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:06.270813 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:06.309850 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:06.348992 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:06.356288 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 08:06:06.377053 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:06.408227 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:06.418718 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:06.503228 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 08:06:06.549103 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:06.601783 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:06.617445 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:06.683298 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:06.713315 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:06.764580 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:06.766835 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:06.810909 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:06.850754 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:06.853189 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:06.919893 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:06.966794 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:07.003554 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:07.062849 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:07.142702 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:07.224658 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:07.226649 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:07.253028 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:07.260267 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:07.288555 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:07.380440 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:07.420364 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:07.428906 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:07.486196 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:07.677816 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:07.692137 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240501 08:06:07.708094 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:07.759076 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:07.783011 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:07.797375 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:07.811856 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:07.822474 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:07.822696 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:07.871438 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:06:07.884177 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:07.914582 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:06:07.915748 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:07.922170 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:07.963147 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:07.977653 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:08.115341 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:08.116955 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:08.145844 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:08.160455 7162 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ffc23756b44d4d588e0a8175075d9bc6 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:06:08.287231 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:08.303056 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:08.332347 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:08.380239 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:08.385479 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:08.387639 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:08.464923 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 08:06:08.472623 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:08.546244 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:08.573161 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:08.573491 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:08.588821 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:08.623392 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:08.736016 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:08.788739 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:08.793207 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:08.819307 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:08.828652 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:08.857426 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:08.941095 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:08.961030 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:09.029963 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:09.040186 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:09.059067 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:09.073448 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 08:06:09.187999 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:09.201395 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:09.216049 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:09.223176 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:09.240154 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:09.264076 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:09.388222 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:09.443022 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:09.476928 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:09.534368 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:09.581166 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:09.603665 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:09.622975 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:09.625371 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:09.657508 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:09.671914 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:09.799762 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:09.817489 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:09.835728 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:09.988945 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:09.993419 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:10.018327 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:10.056497 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 162 similar messages]
W20240501 08:06:10.124564 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:10.126973 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:10.149755 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:10.155202 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:10.248263 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:10.261926 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:10.317214 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:10.383602 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:10.394737 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:10.452461 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:10.472658 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:10.482779 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:10.505290 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:10.632495 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:10.632715 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:10.678054 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:06:10.708211 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:10.749413 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:10.767349 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:10.773175 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:10.793394 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:10.870203 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:10.935206 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:11.009444 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:11.044217 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:11.066252 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:11.093246 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:11.172075 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:11.206305 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:11.227506 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 08:06:11.249258 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:11.264452 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:11.280779 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:11.354702 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:11.360628 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 08:06:11.413957 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:11.440730 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:11.500936 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:11.511739 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:11.515239 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:11.631870 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:11.635792 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:11.648084 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:11.691713 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:11.738047 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:11.766638 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:11.828505 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:11.839659 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:11.855528 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:11.901602 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 08:06:11.999826 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:12.014101 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:12.018703 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:12.023662 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:12.067301 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:12.084656 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:12.147662 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:12.202575 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:12.232960 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:12.311694 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:12.345299 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:12.368682 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:12.519140 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:12.543843 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:12.561928 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:12.631733 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:12.665030 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:12.713764 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:12.718019 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240501 08:06:12.757804 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:12.784518 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:12.827109 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:12.839955 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:12.910291 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:12.929203 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:06:12.939436 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:12.939509 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:12.950866 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:13.053783 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:13.084019 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:13.099330 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:13.099877 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:06:13.234613 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:13.239768 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:13.290602 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:13.322860 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:13.377750 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:13.380987 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:13.435925 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:13.447036 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:13.478365 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:13.498011 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:13.527554 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:13.614167 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:13.670775 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:13.691164 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:13.702415 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 08:06:13.731624 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:13.736304 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:13.752553 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:13.825474 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:13.841939 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:13.858922 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:13.909669 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:13.980322 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:13.990242 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:14.003876 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:14.102440 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:14.223520 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:14.248826 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:14.332520 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:14.343422 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:14.348197 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:14.436666 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 08:06:14.453922 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:14.614773 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:14.625610 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:14.628512 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:14.695883 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:14.732445 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:14.734145 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:14.735163 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:14.745236 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:14.812219 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:14.831821 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:14.837033 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:14.893760 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:14.922211 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:14.926282 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:15.032281 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:15.124779 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 159 similar messages]
W20240501 08:06:15.238361 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:15.247927 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:15.250408 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:15.284101 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:15.355624 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:15.359496 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:15.367930 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:15.381394 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:15.385643 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:15.440217 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:15.444082 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:15.474417 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:15.633423 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:15.663997 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:15.755038 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:15.756516 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:15.756927 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:15.773424 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:15.802361 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:15.857136 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:15.874861 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:15.902443 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:15.902446 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:15.911239 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:15.928272 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:15.942737 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:15.959408 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:16.092468 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:16.161703 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:16.206143 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:16.218405 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:16.300470 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:16.341262 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:16.377141 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:16.398730 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:16.404795 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:16.408356 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:16.446686 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 08:06:16.451412 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 08:06:16.514850 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:16.590950 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:16.758216 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:16.813163 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:16.828030 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:16.959892 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:16.965898 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 08:06:16.984699 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:16.987650 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:17.006979 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:17.023437 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:17.038476 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:17.066004 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:17.163625 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:17.178521 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:17.293457 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:17.305239 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:17.362529 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:17.369488 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:17.423144 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:17.473773 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:17.473794 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:17.509434 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:17.614053 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:17.640056 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:17.716969 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:17.719704 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 08:06:17.760908 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:17.825651 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:17.840381 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:17.876821 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:17.887058 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:17.895025 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:17.902974 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:17.944129 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:06:17.962568 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:18.008587 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:18.060647 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:18.079664 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:18.110077 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:18.112784 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:18.227855 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:18.305898 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:18.310519 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:18.315814 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:18.340250 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:18.396613 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:18.459259 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:18.498059 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:18.507275 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:18.513504 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:18.530550 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:18.540779 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:18.542654 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:18.575258 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:18.678424 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:18.872618 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:18.908258 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:18.913893 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:18.920039 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:18.926723 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:18.964303 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:19.038319 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:19.064936 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:19.064978 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:19.177203 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:19.259333 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:19.386253 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:19.421047 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:19.435810 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 08:06:19.449856 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 08:06:19.453655 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:19.461706 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:19.466029 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:19.488024 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:19.514523 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:19.520581 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:19.556504 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:19.619519 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:19.630671 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:19.762599 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:19.783255 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:19.787693 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:19.813875 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:19.876823 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:19.901113 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:19.976384 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:20.034390 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:20.064419 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:20.115829 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:20.150069 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:20.161659 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 159 similar messages]
W20240501 08:06:20.190953 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:20.286649 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:20.313001 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:20.325924 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:20.344962 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:20.365505 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:20.423492 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:20.430250 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:20.486016 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:20.493902 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:20.525873 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:20.577298 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:20.615561 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:20.629411 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:20.676967 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:20.686744 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:20.726958 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:20.754393 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:20.805426 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:20.883419 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:20.949615 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:20.982134 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:20.991766 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:20.994415 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:21.085494 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:21.135082 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:21.170679 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:21.228471 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:21.278982 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:21.383613 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:21.466035 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:21.498245 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:21.519477 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 08:06:21.521809 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:21.529068 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:21.536360 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:21.540189 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:21.551478 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:21.573120 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:21.598804 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:21.659911 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:21.714998 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:21.802155 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:21.830329 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 08:06:21.899829 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:21.914212 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:21.919606 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:21.954093 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:22.004705 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:22.114725 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:22.146808 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:22.182658 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:22.187381 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:22.192020 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 08:06:22.220844 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:22.291239 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:22.291931 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:22.429775 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:22.475737 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:22.495322 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:22.515465 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:22.553273 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:22.568775 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:22.586475 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:22.656519 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:22.721639 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 08:06:22.729212 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:22.758649 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:22.787231 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:22.831167 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:22.853539 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:22.874950 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:22.877420 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:22.950421 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:06:23.009641 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:23.033592 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:23.053421 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:23.060286 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:23.227151 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:23.227581 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:23.269619 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:23.275457 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:23.286725 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:23.288940 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:23.299685 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:23.317708 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:23.371147 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:23.488976 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:23.504540 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:23.582589 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:23.720772 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:23.764991 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:23.786501 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:23.791167 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:23.845283 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:23.909018 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:23.909484 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:23.921741 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:23.951633 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:24.025641 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:24.044667 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:24.072263 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:24.124675 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:24.172526 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:24.221907 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 08:06:24.224557 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:24.291355 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:24.294560 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:24.315586 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:24.336520 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:24.340423 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:24.380582 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:24.385852 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:24.473582 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:24.493170 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:24.586464 7162 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ffc23756b44d4d588e0a8175075d9bc6 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:06:24.614707 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:24.704787 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:24.727151 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:24.748670 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 08:06:24.863121 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:24.871084 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:24.879848 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:25.034130 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:25.035409 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:25.072919 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:25.092782 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:25.112879 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:25.168339 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:25.181053 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:25.200486 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:25.205058 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 163 similar messages]
W20240501 08:06:25.224326 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:25.298831 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:25.306977 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:25.314236 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:25.350476 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:25.354951 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:25.434310 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:25.455133 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:25.611495 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:25.657169 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:25.657685 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:25.670657 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:25.745353 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:25.771973 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:25.776329 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:25.796116 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:25.803494 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:25.813473 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:25.880854 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:25.926232 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:25.970644 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:25.989094 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:26.058422 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:26.213053 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:26.277897 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:26.299213 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:26.345052 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:26.391883 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:26.417918 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:26.427673 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:26.452512 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:26.455431 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:26.462673 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 08:06:26.523178 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:26.523663 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 08:06:26.537887 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:26.556509 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:26.567672 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:26.619309 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:26.634305 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:26.657878 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:26.778630 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:06:26.779354 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5d18a2918c75467a9fe59979ce8936cd: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:06:26.829658 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:26.899312 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:26.944914 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:26.984164 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:26.998216 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:27.027509 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:27.051031 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:27.059216 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:27.155460 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240501 08:06:27.157725 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:27.161795 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:27.323158 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:27.386296 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:27.473174 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:27.489869 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:27.538123 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:27.542536 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:27.549585 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:27.603430 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:27.665436 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:27.701792 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:27.708326 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:27.725845 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240501 08:06:27.726457 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:27.766211 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:27.768590 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:27.790045 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:27.810428 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:27.823182 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 21d76484f5dd499fb6fc3d534bffb779: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:06:27.834548 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:27.836241 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:27.912832 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:27.966205 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:06:27.998342 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:28.021690 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:28.031565 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:28.064649 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:28.079185 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:28.110321 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:28.129824 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:28.275928 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:28.403956 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:28.417433 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:28.420734 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:28.428043 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:28.559556 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:28.563264 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:28.574635 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:28.590060 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:28.592574 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:28.652309 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:28.804878 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:28.821956 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:28.864519 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:28.896131 7160 catalog_manager.cc:4573] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cc6f7dcc930e4937865d37cb3c3bb3d2: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:06:28.944132 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:29.028832 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:29.083163 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:29.089956 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 08:06:29.090123 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:29.104775 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:29.108965 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:29.112740 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:29.147660 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:29.254308 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:29.273537 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:29.287569 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:29.298403 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:29.355865 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:29.417016 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:29.520310 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 08:06:29.576627 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:29.641767 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:29.652880 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:29.676622 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:29.678267 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:29.699707 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:29.753628 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:29.900728 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:29.918040 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:29.939604 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:29.961153 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:29.981873 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:30.035488 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:30.036459 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:30.037492 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:30.048718 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:30.092677 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:30.094086 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:06:30.155498 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:30.160540 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:30.181123 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:30.185164 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:30.211251 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 160 similar messages]
W20240501 08:06:30.229130 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:30.243502 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:30.289767 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:30.392453 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:30.420646 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:30.457371 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:30.469437 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:30.489946 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:30.654461 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:30.665904 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:30.672122 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:30.716773 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:30.805402 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:30.816598 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:30.847199 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:30.917183 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:30.917618 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:30.947500 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:31.007056 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:31.090466 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:31.130558 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:31.194416 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:31.269840 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:31.281651 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:31.287055 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:31.342396 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:31.367035 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:31.425487 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:31.430828 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:31.437362 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:31.519596 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:31.535701 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 08:06:31.542232 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:31.641798 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:31.646800 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:31.649070 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:31.737156 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:31.797163 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:31.807524 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240501 08:06:31.815510 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:31.840931 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:31.883337 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:31.917688 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:32.017884 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:32.072266 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 08:06:32.095130 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:32.115410 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:32.185431 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:32.260425 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:32.282593 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:32.326406 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:32.362799 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:32.367115 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:32.431010 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:32.434098 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:32.463006 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:32.463973 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:32.590098 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:32.619982 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:32.620790 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:32.622397 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:32.624696 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:32.655722 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:32.659045 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:32.723654 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:32.789791 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:32.807569 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240501 08:06:32.814519 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:32.817589 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:32.855068 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:32.868608 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:32.891690 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:32.922677 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:33.045334 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:06:33.054066 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:33.108011 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:33.142308 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:33.187695 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:33.222568 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:33.245627 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:33.268724 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:33.304404 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:33.322749 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:33.390019 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:33.413347 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:33.434619 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:33.487022 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:33.551554 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:33.622370 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:33.635973 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:33.739682 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:33.774251 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:33.829771 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:33.905205 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:33.923802 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:33.938073 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:33.946475 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:33.995677 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:34.051931 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:34.063331 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:34.128926 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:34.139051 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:34.184792 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:34.211829 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 08:06:34.266626 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:34.271649 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:34.311272 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:34.410955 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:34.439953 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240501 08:06:34.471787 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:34.513031 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:34.531915 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:34.537302 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:34.564874 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:34.690248 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:34.765338 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:34.771775 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:34.789283 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:34.962323 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:34.976116 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:35.026310 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:35.051822 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:35.140648 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:35.156672 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:35.158537 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:35.172122 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:35.223270 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240501 08:06:35.258275 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:35.271307 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:35.275344 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:35.301692 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:35.304634 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:35.315816 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:35.345310 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:35.355620 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:35.416738 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:35.447477 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:35.450803 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:35.521903 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:35.572834 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:35.635483 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:35.755865 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:35.770287 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:35.783288 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:35.784684 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:35.791638 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:35.836011 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:35.961700 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:36.036242 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:36.038653 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:36.068806 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:36.072939 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:36.126309 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:36.126865 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:36.172683 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:36.322824 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:36.352109 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:36.380090 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:36.420359 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:36.443981 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:36.460642 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:36.526393 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:36.575944 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240501 08:06:36.576582 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:36.586584 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:36.600530 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:36.616266 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 08:06:36.701828 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:36.773005 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:36.793789 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:36.856710 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:36.872980 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:36.907374 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:36.947549 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:36.997509 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:37.046978 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 08:06:37.053150 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:37.059075 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:37.069869 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:37.220012 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:37.274784 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:37.431165 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:37.493152 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:37.512974 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:37.515884 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:37.583601 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:37.590260 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:37.592711 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:37.646633 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:37.657897 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:37.669159 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:37.677209 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:37.817955 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240501 08:06:37.858342 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:37.863675 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:37.868561 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:37.868719 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:37.883222 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:37.919073 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:37.986277 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:38.020864 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:38.045519 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:06:38.163293 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:38.164353 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:38.204066 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:38.208281 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:38.211010 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:38.234505 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:38.236332 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:38.247340 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:38.260542 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:38.267328 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:38.268720 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:38.321638 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:38.418490 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:38.466039 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:38.468523 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:38.547457 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:38.699473 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:38.810380 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:38.822415 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:38.853226 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:38.858151 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:38.891772 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:38.975191 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:39.010227 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:39.045404 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240501 08:06:39.149132 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:39.179433 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:39.187389 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:39.238016 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:39.254142 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:39.303596 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:39.308180 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:39.312129 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:39.336386 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:39.375136 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:39.399734 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:39.432034 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:39.435328 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 08:06:39.442215 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:39.510039 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:39.686764 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:39.731068 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:39.829531 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:39.871623 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:39.888864 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:39.921633 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:39.982489 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:40.049336 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:40.088621 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:40.120118 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:40.161259 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:40.194373 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:40.235538 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:40.235821 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:40.246883 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 162 similar messages]
W20240501 08:06:40.247525 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:40.298254 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:40.329663 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:40.333973 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:40.402635 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:40.418202 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:40.441210 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:40.460132 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:40.479750 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:40.528374 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:40.533568 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:40.610421 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:40.679153 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:40.693666 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:40.762804 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:40.798357 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:40.805974 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:40.908675 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:40.909188 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:40.917608 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:40.947161 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:40.950847 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:41.086526 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:41.204092 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:41.222908 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:41.274312 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:41.356155 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:41.392320 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:41.463228 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 08:06:41.498749 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:41.530470 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:41.535614 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:41.575804 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:41.620602 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 08:06:41.621266 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:41.672259 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:41.681463 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:41.684446 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:41.708945 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:41.724498 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:41.755913 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:41.832008 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:41.854027 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:41.859098 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:41.892688 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:41.908807 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:41.950902 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:41.986544 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:42.041705 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 08:06:42.083392 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:42.244674 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:42.277770 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:42.288484 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:42.355337 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:42.419590 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:42.455936 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:42.460592 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:42.511536 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:42.539793 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:42.652132 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:42.674031 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:42.699298 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:42.789180 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:42.834846 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240501 08:06:42.835593 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:42.889788 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:42.891538 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:42.904225 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:42.947486 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:42.948613 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:42.963680 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:43.071213 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:43.098047 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:43.116438 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:43.149101 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:06:43.165613 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:43.185145 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:43.228173 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:43.276376 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:43.283893 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:43.295591 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:43.374714 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:43.415984 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:43.457870 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:43.512691 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:43.567173 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:43.572055 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:43.691643 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:43.733270 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:43.755195 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:43.794559 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:43.856125 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:43.885710 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:43.889288 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:43.919847 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:43.974896 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:43.982789 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:44.054239 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 08:06:44.064765 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:44.077944 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:44.122925 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:44.139024 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:44.183699 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:44.183990 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:44.198163 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:44.209641 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:44.288769 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:44.354853 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:44.363139 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:44.446381 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:44.484434 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:44.548511 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:44.618395 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:44.621480 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:44.626999 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:44.666703 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 08:06:44.691879 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:44.702111 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:44.734771 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:44.832811 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:44.848585 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:44.875303 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:44.949290 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:45.057915 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:45.122745 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:45.165014 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:45.187433 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:45.199072 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:45.248703 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:45.276329 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 160 similar messages]
W20240501 08:06:45.277118 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:45.330080 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:45.374068 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:45.432693 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:45.439481 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:45.511375 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:45.568259 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:45.591782 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:45.594718 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:45.603896 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:45.634488 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:45.741111 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:45.776813 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:45.810637 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:45.845681 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:45.851792 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:45.867391 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:45.900522 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:45.947911 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:46.007687 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:46.016106 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:46.065923 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:46.097098 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:46.215202 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:46.346601 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:46.348162 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:46.450302 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:46.457345 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:46.471721 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:46.481124 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:46.481709 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:46.499332 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:46.503108 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:46.550786 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:46.601084 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:46.609578 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:46.617818 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:46.633345 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:46.662139 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 08:06:46.695247 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 08:06:46.723502 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:46.735010 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:46.751497 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:46.803210 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:46.820726 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:46.913908 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:46.919216 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:46.975735 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:46.979956 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:47.072746 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:47.110038 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:47.185896 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:47.249392 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 08:06:47.276154 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:47.356423 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:47.357379 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:47.375017 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:47.406242 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:47.495496 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:47.522904 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:47.601737 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:47.697305 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:47.721014 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:47.731976 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:47.740350 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:47.762744 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:47.861405 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:47.876415 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240501 08:06:47.947141 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:47.993800 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:48.008877 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:48.015388 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:48.034466 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:48.120486 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:48.173871 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:48.207053 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:06:48.207738 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:48.212136 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:48.251664 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:48.345111 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:48.376608 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:48.380451 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:48.400653 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:48.406082 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:48.410255 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:48.529467 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:48.565579 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:48.629972 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:48.645190 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:48.812536 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:48.831238 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:48.865224 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:48.894045 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:48.895018 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:48.930536 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:48.931982 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:48.956934 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:48.975291 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:49.032315 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:49.035683 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:49.039940 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:49.092638 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:49.138295 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:49.181536 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:49.234268 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:49.239473 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 08:06:49.244267 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:49.246366 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:49.399344 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:49.411137 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:49.486259 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:49.498761 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:49.506165 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:49.555742 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:49.559861 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:49.653499 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:49.756887 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:49.774931 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:49.784885 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:49.796434 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:49.832648 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:49.871237 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:49.908768 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 08:06:50.040133 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:50.106851 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:50.113183 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:50.139318 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:50.169502 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:50.315078 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:50.371448 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 163 similar messages]
W20240501 08:06:50.372172 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:50.386662 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:50.420118 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:50.455896 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:50.503065 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:50.514869 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:50.552695 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:50.569437 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:50.591446 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:50.647054 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:50.698048 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:50.775691 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:50.840705 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:50.849685 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:50.860248 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:50.917160 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:50.924846 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:50.955646 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:50.988193 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:50.988850 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:51.093379 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:51.142045 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:51.166729 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:51.173848 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:51.238881 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:51.306057 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:51.320299 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:51.335162 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:51.460227 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:51.493516 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:51.496578 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:51.525076 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:51.560675 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:51.565322 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:51.622256 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:51.628664 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:51.646678 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:51.663947 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 08:06:51.685931 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:51.725718 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:51.773849 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:51.815271 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 08:06:51.819415 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:51.835501 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:51.840169 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:51.870180 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:51.872233 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:51.976060 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:52.083444 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:52.148422 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:52.223253 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:52.283219 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:52.290176 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 08:06:52.296140 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:52.383698 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:52.416236 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:52.428350 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:52.482152 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:52.588714 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:52.621346 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:52.662649 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:52.703274 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:52.712723 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:52.758984 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:52.879202 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:52.931385 7250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 08:06:53.003595 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:53.031495 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:53.038715 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:53.058702 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:53.075992 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:53.080857 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:53.110980 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:53.191309 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:53.203378 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:53.208945 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:53.214547 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:06:53.215159 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:53.233659 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:53.244186 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:53.306346 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:53.350674 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:53.382436 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:53.438942 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:53.457140 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:53.484740 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:53.523444 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:53.527014 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:53.574203 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:53.628943 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:53.662846 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:53.697580 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:53.818647 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:53.833810 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:53.915685 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:53.953291 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:53.973691 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:53.983021 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:54.027194 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:54.040356 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:54.048125 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 08:06:54.096418 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:54.157544 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:54.232180 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:54.233584 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:54.279160 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:54.333233 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:54.336105 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:54.384614 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:54.394253 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:54.413189 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:54.456065 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:54.476128 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:54.482865 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:54.553381 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:54.568069 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:54.664144 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:54.806087 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:54.879092 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:54.883836 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:54.889281 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:54.913775 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:54.931578 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:54.954393 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 08:06:55.058506 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:55.061744 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:55.089763 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:55.157783 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:55.393842 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240501 08:06:55.394109 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:55.449198 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:55.449599 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:55.467999 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:55.479804 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:55.501109 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:55.533207 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:55.551748 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:55.672071 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:55.672829 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:55.701932 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:55.772766 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:55.774847 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:55.864192 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:55.883687 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:55.920658 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:55.942458 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:55.991583 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:55.992061 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:56.014920 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:56.032239 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:56.044559 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:56.099143 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:56.103247 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:56.110623 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:56.127401 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:56.173769 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:56.176337 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:56.329424 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:56.382279 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:56.456960 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:56.463076 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:56.524101 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:56.562536 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 08:06:56.568605 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:56.599197 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:56.722376 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 08:06:56.728355 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:56.734521 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:56.737428 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:56.746553 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:56.756687 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:56.781476 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:56.794917 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:56.817132 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:56.895545 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:56.945385 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:57.033196 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:57.066709 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:57.084137 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:57.173666 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:57.197358 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:57.221678 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:57.256880 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:57.265740 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:57.367646 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:57.372072 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:57.374644 7162 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ffc23756b44d4d588e0a8175075d9bc6 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:06:57.389415 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:57.429437 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:57.503734 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 08:06:57.686105 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:57.686587 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:57.690042 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:57.763159 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:57.842831 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:58.011943 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:58.072202 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:58.072592 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:58.085103 7250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 08:06:58.095659 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:58.097384 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:58.133024 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:58.135442 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:58.177896 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:58.183218 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:58.223666 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:58.226648 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:58.266211 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:58.271703 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240501 08:06:58.305258 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:58.323100 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:58.385246 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:58.448871 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:58.495815 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:58.515019 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:58.516281 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:58.520015 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:58.520184 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:58.526669 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:58.637252 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:58.638999 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:58.677542 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:58.706182 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:58.769054 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:58.784699 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:58.798151 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:58.801882 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 08:06:58.916105 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:58.925398 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:58.994571 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:59.055833 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:59.078269 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:59.090206 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:59.194454 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:59.223649 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:06:59.272380 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:59.273371 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:59.366071 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:59.393746 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:59.423717 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:59.476491 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:59.477582 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:59.536546 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:59.555362 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:59.641971 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:59.659524 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:59.669487 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:06:59.794222 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:06:59.802810 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:59.823464 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:59.918592 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:06:59.936769 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:59.994732 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:06:59.995246 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:00.054091 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 08:07:00.113794 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:00.150868 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:00.212035 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:00.259552 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:00.333900 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:00.381475 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:00.405911 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:00.483793 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:00.493247 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 163 similar messages]
W20240501 08:07:00.493793 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:00.516477 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:00.557686 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:00.576009 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:00.654664 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:00.659183 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:00.745916 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:00.814389 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:00.844154 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:00.883373 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:00.903752 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:00.906145 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:00.921169 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:00.981245 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:07:01.078223 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:01.100998 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:01.104282 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:07:01.142666 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:01.168587 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:01.173935 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:01.185954 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:01.187695 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 08:07:01.209192 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:01.209298 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:01.210651 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:01.258467 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:01.296473 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:01.401916 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:01.510051 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:01.537902 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:01.596956 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:01.629794 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:01.636466 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:01.665238 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:01.668721 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:01.689733 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:01.708273 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:01.715528 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:01.728425 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 08:07:01.794188 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:01.904536 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:02.043319 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:02.082595 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:02.092872 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:02.143522 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:02.166715 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:02.184260 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:02.264127 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:02.295130 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:02.305546 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:02.418038 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:02.482460 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:02.556864 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:02.596740 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 08:07:02.657686 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:02.659533 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:02.683722 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:02.726338 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:02.726979 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:02.729602 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:02.745965 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:02.938443 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:02.976956 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:03.051096 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:03.075147 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:03.092214 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240501 08:07:03.115332 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:03.159406 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:03.160493 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:03.199950 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:03.209725 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:03.249197 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:03.309003 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:07:03.332331 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:03.339512 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:03.357715 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:03.360613 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:03.432595 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:03.452024 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:03.459688 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:03.487048 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:03.595113 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:03.625367 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:03.625792 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:03.630631 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:03.643275 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:03.663412 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:03.678833 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:03.717864 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:03.734613 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:07:03.766357 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:03.788580 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 08:07:03.797359 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:03.851047 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:03.880086 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:03.967598 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:04.003142 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:04.034829 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:04.073318 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:04.101477 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:04.109578 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:04.113965 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:04.241355 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:04.259001 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:04.348690 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:04.530858 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:04.550199 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:04.566980 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:04.592777 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:04.670632 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:04.689718 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:04.705483 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:04.783772 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:04.910598 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:04.939646 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:04.987268 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:05.024523 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:05.106974 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:05.107549 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:05.141026 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:05.169090 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:05.194656 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:05.222728 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:05.258769 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 08:07:05.259807 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:05.287740 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:05.370494 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:05.506350 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 159 similar messages]
W20240501 08:07:05.507083 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:05.553112 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:05.604415 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:05.654187 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:05.669802 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:05.688689 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:05.773263 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:05.783691 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:05.849664 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:05.880082 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:05.885329 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:05.894650 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:05.960283 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:05.984342 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:06.018008 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:06.050323 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:06.057569 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:06.065768 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:06.075824 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:06.107429 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:06.131448 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:06.146426 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:06.161572 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:06.222432 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 08:07:06.242334 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:06.267639 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:06.272871 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:06.325481 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:06.351269 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:06.367210 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:07:06.467897 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:06.475147 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:06.490231 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:06.499496 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:06.575261 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:06.682346 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:06.716099 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:06.721694 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:06.733575 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240501 08:07:06.823889 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:06.850874 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:06.863482 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:06.954447 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:06.995441 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:07.058967 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:07.093756 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:07.099285 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:07.176765 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:07.311067 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:07.362840 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:07.435034 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:07.444195 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:07.478554 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:07.521577 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:07.561161 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:07.589293 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:07.590411 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:07.671087 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:07.718950 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:07.719348 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:07.732213 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:07.801244 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:07.842167 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:07.925326 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:07.941062 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:08.018556 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:08.022074 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:08.031073 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 08:07:08.100566 7250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 08:07:08.149856 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:08.158959 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:08.183646 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:08.247457 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:08.345220 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:08.355285 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:08.363281 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:08.374105 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:08.386941 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:08.425920 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:07:08.426537 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:08.477952 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:08.481772 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:08.563108 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:08.566119 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:08.584568 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:08.584568 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:08.597244 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:08.676067 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:08.738142 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:08.741276 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:08.746520 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:08.749775 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:07:08.761162 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:08.815732 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 08:07:08.817441 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:08.835765 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:08.891647 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:08.916996 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:08.975898 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:08.982429 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:08.984831 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:09.040517 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:09.074967 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:09.111157 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:09.132649 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:09.173673 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:09.316848 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:09.481618 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:09.559060 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:09.595038 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:09.596843 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:09.661008 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:09.700495 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:09.734928 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:09.794714 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:09.794799 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:09.917500 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:09.949131 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:10.017429 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:10.021373 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:10.063759 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:10.119666 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:10.119999 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:10.128625 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:10.135165 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:10.433198 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:10.442382 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:10.461305 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:10.468933 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:10.476439 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:10.477562 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:10.495432 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:10.498540 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:10.512719 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 08:07:10.523228 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:10.591971 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:10.683790 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 165 similar messages]
W20240501 08:07:10.684279 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:10.692014 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:10.715598 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:10.790609 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:10.846542 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:10.859894 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:10.860276 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:10.894601 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:10.911473 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:10.924754 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 08:07:10.967790 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:11.012856 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:11.035524 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:11.087661 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:11.100493 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:11.140381 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:11.156765 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:11.204373 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:07:11.214231 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:11.228086 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:11.258430 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:11.349642 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:11.362551 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:11.386389 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:11.433794 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:11.456568 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:11.466051 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:11.527169 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:11.538729 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:11.568728 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:11.637392 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:11.648978 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:11.705904 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:11.706604 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:11.762578 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 08:07:11.883764 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:11.919368 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:11.992481 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:12.138940 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:12.238991 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:12.263150 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:12.296149 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:12.335608 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:12.411651 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:12.427814 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:12.457429 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:12.477905 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:12.497879 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:12.553915 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:12.587528 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:12.611397 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:12.626947 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:12.736712 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:12.807508 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:12.894160 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:12.908767 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:12.960127 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:12.979890 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:13.012064 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:13.014467 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:13.050065 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 08:07:13.066798 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:13.069765 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:13.080816 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:13.083163 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:13.194080 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240501 08:07:13.213243 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:13.235877 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 08:07:13.244204 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:13.311733 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:13.321789 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:13.385828 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:13.415814 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:13.432526 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:13.438354 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:13.446154 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:13.454509 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:13.465914 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:13.528587 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:07:13.566907 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:13.587460 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:13.612694 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:13.692919 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:13.828765 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:13.830282 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:13.855558 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:13.872435 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:07:13.920424 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:13.930016 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:13.969182 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:14.016532 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:14.027433 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:14.030701 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:14.101125 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:14.101114 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:14.169395 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:14.212529 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:14.213143 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:14.299844 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:14.307765 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:14.406733 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:14.517709 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:14.558720 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:14.654685 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:14.814944 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:14.823141 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:14.831661 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:14.922173 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:14.941906 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:14.954288 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:14.963500 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:14.982134 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:14.985350 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:15.113183 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:15.208262 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:15.245055 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:15.261940 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:15.262290 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:15.264572 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:15.397097 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:15.429363 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:15.449656 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:15.451061 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:15.516701 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:15.589696 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:15.601889 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:15.669421 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:15.697376 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240501 08:07:15.709726 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:15.710338 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 08:07:15.712396 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:15.724328 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:15.751283 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:15.762534 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:15.794775 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:15.800383 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:15.806329 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:15.836239 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:15.854657 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:15.892601 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:16.000088 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 08:07:16.025516 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:16.044095 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:16.058054 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:16.070950 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:16.106920 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:16.173312 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:16.259011 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:16.293957 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:16.304209 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:16.464134 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:16.490692 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:16.549468 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:16.551323 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:16.575416 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:16.595420 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:16.691212 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:16.711472 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:16.721879 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:16.728957 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:16.742331 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:16.766083 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:16.771027 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 08:07:16.776185 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:16.804056 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:16.881790 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:17.180263 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:17.184850 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:17.186623 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:17.190583 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:17.235960 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:17.244410 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:17.305189 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:17.398384 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:17.451557 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:17.463577 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:17.581765 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:17.585400 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:17.609426 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:17.653475 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:17.666580 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:17.711508 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:17.874717 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:17.875672 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:17.905943 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:17.913590 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:18.005062 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:18.038189 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:18.041281 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:18.071194 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:18.113862 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:18.129280 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:18.167363 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:18.168326 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:18.184563 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:18.186930 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:18.208686 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 08:07:18.213788 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 08:07:18.317087 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:18.343214 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:18.344904 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:18.367727 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:18.418531 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:18.469506 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:18.491009 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:18.528717 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:18.543692 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:07:18.544302 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:18.564975 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:18.565757 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 08:07:18.610922 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:18.641875 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:18.692523 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:18.706800 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:18.734283 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:18.887096 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:18.954324 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:18.987361 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:19.073107 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:19.082056 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:19.099956 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:19.105162 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:19.141346 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:19.290017 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:19.304958 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:19.324872 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:19.331843 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:19.331990 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:19.378548 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:19.408052 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:19.437799 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:19.487457 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:19.497124 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:19.594760 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:19.672600 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:19.693657 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:19.718183 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:19.728255 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:19.765453 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:19.789273 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:19.851194 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:19.871258 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:20.052899 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:20.096638 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:20.191208 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:20.232995 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:20.276162 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:20.287035 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:20.289656 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:20.367864 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:20.419446 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:20.438663 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:20.446718 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:20.530088 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:20.535394 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:20.558799 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:20.652069 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:20.693446 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:20.714807 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 159 similar messages]
W20240501 08:07:20.730939 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:20.763669 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:20.779738 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:20.799638 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:20.806532 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:20.813855 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:20.838809 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:20.902231 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 08:07:20.918910 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:20.923561 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:20.934649 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:20.986503 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:07:20.989001 7164 maintenance_manager.cc:419] P 6aa238de40d94e5f9898ac424a4d1f68: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033384
I20240501 08:07:20.991868 7163 maintenance_manager.cc:643] P 6aa238de40d94e5f9898ac424a4d1f68: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.001s sys 0.000s Metrics: {"bytes_written":2680,"delete_count":0,"lbm_write_time_us":120,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":66,"threads_started":1,"update_count":39,"wal-append.queue_time_us":153}
I20240501 08:07:20.992225 7164 maintenance_manager.cc:419] P 6aa238de40d94e5f9898ac424a4d1f68: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.177143
W20240501 08:07:20.997443 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240501 08:07:20.999478 7163 maintenance_manager.cc:643] P 6aa238de40d94e5f9898ac424a4d1f68: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.005s sys 0.002s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":15944,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":164,"dirs.run_cpu_time_us":79,"dirs.run_wall_time_us":80,"lbm_read_time_us":225,"lbm_reads_lt_1ms":17,"lbm_write_time_us":229,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":139,"threads_started":2,"update_count":39}
W20240501 08:07:21.036525 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:21.095289 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:21.106132 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:21.145304 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:21.195117 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:21.220847 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:21.231346 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:21.308159 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:21.337579 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:21.351794 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:21.421201 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:21.555183 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:21.575152 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:21.613866 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:21.674109 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:21.707191 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:21.771883 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240501 08:07:21.778713 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:21.805140 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:21.925879 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:21.939026 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:21.943758 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:21.955040 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:21.994464 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:22.026553 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:22.052400 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:22.100174 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:22.115883 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:22.118564 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:22.200181 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:22.200791 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:22.250823 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:22.332221 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:22.362632 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:22.489390 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:22.575861 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:22.632645 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:22.699476 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:22.766106 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:22.797502 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:22.839042 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:22.864696 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:22.881583 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:22.900056 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:22.915378 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:23.052886 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:23.071419 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:23.110641 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:23.185956 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:23.209093 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:23.216861 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 08:07:23.217588 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:23.235738 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:23.250267 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:23.255695 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:23.297812 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:23.299661 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:23.366868 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:23.367697 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:23.374554 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:23.400502 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:23.426093 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:23.460978 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 08:07:23.462656 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:23.494019 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 08:07:23.652899 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:23.666210 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:23.701377 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:23.720775 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:23.720850 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:23.758888 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:07:23.804942 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:23.830865 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:23.882674 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:23.887650 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:24.003666 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:24.036586 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:24.145063 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:24.165441 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:24.231514 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:24.247329 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:24.254907 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:24.256757 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:24.265951 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:24.271579 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:24.406978 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:24.461551 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:24.483438 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:24.510239 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:24.549556 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:24.552012 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:24.646967 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:24.688788 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:24.746588 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:24.760774 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:24.839602 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:24.840849 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:24.973150 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:25.013669 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:25.053551 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:25.063881 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:25.125941 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:25.159507 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:25.320307 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:25.339910 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:25.354806 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:25.363020 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:25.390848 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:25.413444 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:25.432924 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:25.454013 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:25.540710 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:25.586076 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:25.637241 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:25.683127 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 08:07:25.731055 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240501 08:07:25.731699 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:25.750793 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:25.782145 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:25.784675 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:25.800889 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 08:07:25.824239 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:25.856069 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:25.857759 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:25.859244 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:25.876123 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:25.970041 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:26.045629 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:26.046123 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:26.087087 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:26.117810 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:26.141120 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:26.145148 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:26.171160 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:26.276978 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:26.384891 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:26.471966 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:26.513263 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:26.581728 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:26.632697 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:26.632774 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:26.666322 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:26.669802 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:26.678436 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:26.703759 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:26.715696 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:26.812717 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 08:07:26.813527 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:26.819044 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20240501 08:07:26.820106 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5d18a2918c75467a9fe59979ce8936cd: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:07:26.822242 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:26.907788 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:26.907830 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:26.915498 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:26.927547 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:26.952750 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:27.095577 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:27.167040 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:27.176162 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:27.182032 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:27.186007 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:27.268817 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:27.273885 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:27.305387 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:27.378544 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:27.474828 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:27.502835 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:27.563588 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:27.572578 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:27.659330 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:27.704965 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:27.724499 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:27.743965 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:27.744552 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:27.754447 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:27.810532 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:27.864796 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:27.871032 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 21d76484f5dd499fb6fc3d534bffb779: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:07:27.917196 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 08:07:27.947211 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:28.051715 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:28.065146 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:28.128278 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:28.131207 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:28.175689 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 08:07:28.288575 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:28.307267 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:28.318019 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240501 08:07:28.353703 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:28.411073 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:28.438757 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:28.464262 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:28.491083 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:28.511520 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:28.519618 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:28.558548 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:28.595618 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:28.617888 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:28.644690 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:28.735157 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:28.746049 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:28.818532 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:28.849094 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:28.904572 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 85 similar messages]
W20240501 08:07:28.905254 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:28.945180 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:28.946677 7160 catalog_manager.cc:4573] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cc6f7dcc930e4937865d37cb3c3bb3d2: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:07:29.005131 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:29.051201 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:29.082559 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:29.086812 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:29.089610 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:29.098951 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:29.138193 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:29.206926 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:29.327149 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:29.367187 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:29.371698 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:29.376446 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:29.420936 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:29.433890 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:29.485965 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:29.491125 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:29.499166 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:29.528646 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:29.604589 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:29.640300 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:29.651984 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:29.675680 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:29.880733 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:29.909178 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:30.061970 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:30.091629 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:30.135988 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:07:30.157127 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:30.161302 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:30.165952 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:30.287189 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:30.316954 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:30.323491 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:30.346246 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:30.379338 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:30.389847 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:30.398826 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:30.405476 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:30.409329 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:30.440785 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 08:07:30.485342 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:30.589511 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:30.688820 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:30.692581 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 08:07:30.707741 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:30.769672 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:30.782413 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 162 similar messages]
W20240501 08:07:30.783043 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:30.832635 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:30.854910 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:30.896631 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:30.912580 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:30.949360 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:30.977404 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:31.037719 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:31.039316 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:31.080700 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:31.087910 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:31.168709 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:31.283841 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:31.302735 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:31.303777 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:31.415102 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:31.415308 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:31.437496 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:31.439460 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:31.559399 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:31.566718 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:31.572149 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:31.595654 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:31.668813 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:31.683579 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:31.704444 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:31.757900 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:31.814502 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:31.838773 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 08:07:31.839505 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:31.879227 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:31.928514 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:31.940928 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:31.942855 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:31.982867 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:32.033317 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:32.089875 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:32.096752 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:32.114315 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:32.164113 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:32.324600 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:32.472507 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:32.489384 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:32.511834 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:32.545079 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:32.586509 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:32.611956 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:32.619127 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:32.816352 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:32.820415 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:32.851200 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:32.886269 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:32.890458 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 08:07:32.891621 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:32.938261 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:32.957127 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:33.020109 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:33.072484 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:33.083683 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:33.126792 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:33.146260 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:33.150564 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:33.253291 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:33.361583 7250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 08:07:33.383630 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 08:07:33.396116 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:33.420271 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:33.454233 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:33.469836 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:33.496191 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:33.499853 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:33.569080 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:33.574294 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:33.576917 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:33.645923 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:33.650251 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:33.652712 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:33.662295 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:33.821838 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:33.890944 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:33.892555 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:33.924275 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:07:33.968407 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:34.003911 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:34.004175 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:34.098410 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:34.154201 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:34.175426 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:34.213629 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:34.217152 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:34.226325 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:34.230859 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:34.236617 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:34.256538 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:34.319940 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:34.334271 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:34.334478 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:34.373667 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:34.390381 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:34.393913 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:34.421216 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:34.472488 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:34.516127 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:34.696813 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:34.759073 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:34.850275 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:34.850325 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:34.859463 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:34.926245 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:34.975171 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:35.016178 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:35.175632 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:35.196089 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 08:07:35.232139 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:35.282821 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:35.342567 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:35.406105 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:35.428045 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:35.451967 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:35.453389 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:35.587810 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:35.640630 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:35.657704 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:35.673803 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 08:07:35.696471 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:35.728766 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:35.755390 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:35.788136 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:35.804443 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:35.810415 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 162 similar messages]
W20240501 08:07:35.821674 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:35.840793 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:35.937445 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:35.948309 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:35.962373 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:36.031240 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:36.047672 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:36.129320 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:36.171888 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:36.194499 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:36.203167 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:36.268422 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:36.307456 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:36.374459 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:36.391186 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:36.423717 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:36.450568 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:36.460526 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:36.537003 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:36.542903 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:36.579062 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:36.608717 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:36.672709 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:36.693080 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:36.714080 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:36.760634 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:36.768615 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:36.822867 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:36.858301 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:36.862139 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240501 08:07:36.914208 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:36.914189 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:36.914701 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:36.982445 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:37.015528 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:37.032524 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:37.060837 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:37.201752 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:37.204484 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:37.216320 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:37.367868 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:37.376068 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:37.410359 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:37.424858 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:37.434069 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:37.498760 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:37.625392 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:37.678880 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:37.724790 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:37.804661 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:37.855895 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 08:07:37.913199 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:37.939558 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:37.973479 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:38.001178 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:38.001744 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:38.004066 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:38.053412 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:38.119813 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:38.138008 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:38.218653 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:38.249262 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:38.255270 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 08:07:38.300468 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:38.334482 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:38.366858 7250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240501 08:07:38.378394 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:38.420419 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:38.454869 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:38.455090 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:38.480165 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:38.490253 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:38.499694 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:38.501188 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:38.565523 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:38.569056 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:38.592607 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:38.712570 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:38.839615 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:38.843942 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:38.900422 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:38.924085 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:38.935798 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:07:38.949093 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:39.018154 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:39.046316 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:39.063848 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:39.071727 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:39.095077 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:39.129659 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:39.247617 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:39.271049 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:39.342546 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:39.350198 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:39.359642 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:39.363787 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:39.444656 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:39.475551 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:39.480031 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:39.495352 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:39.610276 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:39.610370 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:39.672711 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:39.724959 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:39.820690 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:39.822196 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:39.826257 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:39.943692 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:40.001067 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:40.045449 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 08:07:40.047297 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:40.063783 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:40.186800 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:40.254756 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:40.277997 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:40.293449 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:40.442728 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:40.483299 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:40.516937 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:40.575517 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:40.610831 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:40.618963 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:40.641263 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:40.651213 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:40.653708 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:40.675156 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:40.686281 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 08:07:40.716377 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:40.806851 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:40.828536 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240501 08:07:40.889245 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:40.909399 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:40.914067 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:40.950304 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:40.972118 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:41.031985 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:41.062439 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:41.076434 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:41.120450 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:41.122030 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:41.136232 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:41.183087 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:41.198868 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:41.273243 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:41.307159 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:41.423976 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:41.495653 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:41.526556 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:41.541805 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:41.575646 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:41.587440 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:41.623075 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:41.648556 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:41.653551 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:41.696399 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:41.716992 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:41.733073 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:41.756855 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:41.812795 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:41.857921 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:41.958432 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:41.999708 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 08:07:42.000305 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:42.006578 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:42.030512 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:42.042531 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:42.085458 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:42.125648 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:42.142707 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:42.179009 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:42.219267 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:42.266676 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:42.302359 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:42.441359 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:42.469966 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 08:07:42.491895 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:42.494493 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:42.643096 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:42.686326 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:42.712684 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:42.744607 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:42.831537 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:42.918061 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:42.920010 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:43.011341 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:43.020694 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:43.036934 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:43.058300 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:43.168219 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:43.203137 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:43.205785 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 08:07:43.260370 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:43.275631 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:43.302997 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:43.336510 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:43.416079 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 08:07:43.445106 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:43.453364 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:43.530969 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:43.545408 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:43.555991 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:43.558831 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:43.640605 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:43.648452 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:43.664238 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:43.675504 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:43.688546 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:43.707289 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:43.710759 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:43.723132 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:43.799664 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:43.870203 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:43.900007 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:43.940351 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:07:43.967988 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:43.981833 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:44.018306 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:44.025136 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:44.033856 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:44.106463 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:44.205722 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:44.212076 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:44.239953 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:44.251441 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:07:44.270794 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:44.363675 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:44.402957 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:44.432597 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:44.451733 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:44.460682 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:44.490965 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:44.567503 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:44.571151 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:44.576845 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:44.678216 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:44.744381 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:44.780761 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:44.821655 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:44.877938 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:44.959652 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:45.107455 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:45.112107 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:45.125015 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 08:07:45.252872 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:45.319998 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:45.343801 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:45.363946 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:45.364542 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:45.421903 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:45.427632 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:45.448033 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:45.520545 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:45.573359 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:45.598340 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:45.626205 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:45.652276 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:45.846161 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:45.856315 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:45.884124 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 160 similar messages]
W20240501 08:07:45.896124 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:45.930220 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:45.934660 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:45.952148 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:45.972543 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:45.978714 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:45.988417 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:45.997828 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 08:07:46.009619 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:46.062793 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:46.087378 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:46.107724 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:46.332557 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:46.364847 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:46.375973 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:46.382673 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:46.418603 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:46.429477 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:46.486625 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:46.488672 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:46.528080 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:46.530481 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:46.534145 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:46.581130 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:46.614012 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:46.653730 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:46.716674 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:46.764262 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:46.776553 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:46.779002 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:46.859218 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:46.925654 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:46.947892 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:46.966501 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:47.035688 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 08:07:47.085239 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:47.085373 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:47.131224 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:47.170037 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:47.253616 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:47.266439 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:47.271929 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:47.286916 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:47.389047 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:47.391438 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:47.420984 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:47.479466 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:47.665498 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:47.708514 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:47.709050 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:47.746481 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 08:07:47.806294 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:47.857008 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:47.857795 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:47.919901 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:48.015174 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:48.080871 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:48.085937 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:48.185159 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:48.189338 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:48.194543 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:48.215507 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:48.253027 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:48.254910 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:48.261516 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:48.289217 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:48.396698 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 08:07:48.418850 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:48.480427 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:48.550895 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 08:07:48.551529 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:48.579147 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:48.616466 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:48.628177 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:48.689544 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:48.727609 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:48.784510 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:48.788872 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:48.792481 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:48.795099 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:48.819761 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:48.824987 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:48.876433 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:48.879262 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:48.903297 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:48.946333 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:49.011082 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:49.026288 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:49.035401 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:07:49.053030 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:49.073935 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:49.196635 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:49.212774 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:49.225458 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:49.311316 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:49.359665 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:49.436293 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:49.486168 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:49.508798 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:49.516736 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:49.584422 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:49.631511 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:49.682883 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:49.700210 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:49.764509 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:49.774046 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:49.830593 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:49.863791 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:49.871723 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:49.892948 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:49.903152 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:49.907297 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:50.094511 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:50.095008 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 08:07:50.103964 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:50.118319 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:50.278731 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:50.280884 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:50.389312 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:50.437670 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:50.448133 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:50.477126 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:50.542693 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:50.593132 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:50.601788 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:50.639516 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:50.666131 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:50.745069 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:50.752416 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:50.813727 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 08:07:50.884320 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240501 08:07:50.894127 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:50.914446 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:50.918900 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:50.926044 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:50.929317 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:51.002118 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:51.042522 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:51.056679 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:51.114082 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:51.161157 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:51.177449 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:51.327037 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:51.333725 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:51.405313 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:51.416254 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:51.455049 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:51.471753 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:51.492822 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:51.525153 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:51.631227 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:51.662660 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:51.667034 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:51.698180 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:51.726047 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:51.763609 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:51.813970 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:51.832367 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:51.879951 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:51.913520 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:51.928843 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:51.952477 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:51.996047 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:52.025250 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:52.053476 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 08:07:52.073338 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:52.222456 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:52.249678 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:52.274165 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:52.284425 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:52.316128 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:52.409078 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:52.439280 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:52.481446 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:52.519348 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:52.529956 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:52.605449 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:52.696053 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:52.778548 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 08:07:52.792884 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:52.824064 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:52.843937 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:52.907603 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:52.976979 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:52.977644 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:52.986552 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:53.067479 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:53.092214 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:53.138554 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:53.239145 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:53.276489 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:53.323894 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:53.363204 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:53.409075 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:53.414830 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:53.429474 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240501 08:07:53.447264 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:53.464223 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:53.528967 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:53.545193 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:53.551353 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 08:07:53.606757 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:53.610242 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:53.631870 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:53.679033 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:53.681336 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:53.693795 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:53.703358 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:53.860190 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:53.917131 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:53.972827 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:54.029781 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:54.074878 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:54.078835 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:07:54.094293 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:54.139955 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:54.156701 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:54.230820 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:54.231313 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:54.238710 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:54.252382 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:54.282193 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:54.296051 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:54.325371 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:54.353830 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:54.473165 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:54.527755 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:54.562927 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:54.573823 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:54.690627 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:54.720168 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:54.766125 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:54.819391 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:54.853715 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:54.885164 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:54.888150 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:54.916805 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:54.982174 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:54.984181 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:55.033080 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:55.035270 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:55.171752 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:55.197098 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:55.275810 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240501 08:07:55.290278 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:55.321641 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:55.440739 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:55.464612 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:55.473101 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:55.509531 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:55.564077 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:55.609803 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:55.621866 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:55.653926 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:55.680836 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:55.690263 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:55.739840 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:55.755309 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:55.837870 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:55.907959 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:55.914251 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240501 08:07:55.914920 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:55.934484 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:55.987002 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:56.031402 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:56.079339 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:56.091921 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:56.118412 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:56.163601 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 08:07:56.231882 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:56.273347 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:56.309006 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:56.336117 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:56.351826 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:56.372366 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:56.502744 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:56.587878 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:56.612154 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:56.634449 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:56.778690 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:56.800475 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:56.841262 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:56.857707 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:56.872767 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:56.887392 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:56.930855 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:56.941562 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:56.954005 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:56.958503 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:56.990640 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:57.002799 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:57.025367 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:57.054361 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:57.057963 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 08:07:57.059725 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:57.159497 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:57.318378 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:57.328522 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:57.348874 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:57.370579 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:57.377910 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:57.380874 7162 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ffc23756b44d4d588e0a8175075d9bc6 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:07:57.401752 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:57.409169 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:57.433144 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:57.530025 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:57.551940 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:57.552453 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:57.652686 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 08:07:57.727481 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:57.731721 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:57.799433 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:57.826675 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:57.883255 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:57.984629 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:58.016338 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:58.017792 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:58.066982 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:58.101047 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:58.185948 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:58.224854 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:58.258832 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:58.275269 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 08:07:58.285367 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:58.287890 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:58.299378 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:58.312850 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:58.328670 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:58.344755 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:58.551529 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:58.592123 7250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 08:07:58.621646 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:58.624699 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:58.703600 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:58.788363 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:58.835008 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:58.840991 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:58.844708 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:58.857503 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:58.917039 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:58.970566 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:58.986073 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:59.062878 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:59.090699 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:07:59.109002 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:07:59.211036 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:59.215044 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:59.220525 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:59.270326 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:59.321806 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:59.324127 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:59.406528 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:59.445372 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:07:59.479247 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:59.510154 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:59.512082 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:59.531718 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:59.555662 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:59.593637 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:59.648618 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:07:59.675190 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:59.694232 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:59.698164 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:59.698182 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:07:59.776180 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:59.783402 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:59.789173 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:07:59.879048 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:00.002326 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:00.013877 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:00.074018 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:00.135883 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:00.139763 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:00.156633 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:00.211535 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:00.412863 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 08:08:00.434072 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:00.465777 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:00.467161 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 08:08:00.488317 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:00.558449 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:00.588407 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:00.651023 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:00.655086 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:00.667724 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:00.685971 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:00.714135 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:00.716887 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:00.739432 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:00.747246 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:00.780095 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:00.818707 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:00.843222 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:00.848961 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:01.000130 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:01.000945 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 159 similar messages]
W20240501 08:08:01.005471 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:01.116286 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:01.235294 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:01.266587 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:01.322583 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:01.323329 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:01.334043 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:01.353137 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:01.429333 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:01.436794 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:01.459841 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:01.539345 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:01.599537 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:01.709828 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:01.762679 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:01.781347 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:01.804479 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:01.809185 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:01.813684 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:01.875383 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:01.876845 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:01.937314 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:01.975840 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:02.067806 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 08:08:02.068444 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:02.106989 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:02.136562 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:02.166760 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:08:02.166993 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:02.192415 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:02.192559 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:02.261790 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:02.269570 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:02.274808 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:02.318651 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:02.347028 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:08:02.431762 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:02.463037 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:02.492846 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:02.508662 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:02.622574 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:02.688156 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:02.764848 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 08:08:02.801200 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:02.808979 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:02.812350 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:02.875233 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:02.926635 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:02.935555 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:02.937453 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240501 08:08:03.041430 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:03.070947 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:03.103217 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:03.112084 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:03.123219 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:03.131989 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:03.177893 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:03.181473 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:03.264847 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:03.270891 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:03.384930 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:03.466029 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:03.492412 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:03.586958 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:03.592610 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 08:08:03.605271 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:03.655331 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:03.717164 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:03.729738 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:03.752347 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:03.765683 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:03.815955 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:03.948390 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:03.962767 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:04.013029 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:04.087997 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:04.093143 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:04.114662 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:04.183313 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:04.206022 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:04.213678 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:08:04.214354 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:04.278002 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:04.349066 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:04.356794 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:04.378842 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:04.388978 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:04.454571 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:04.464732 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:04.566277 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:04.572410 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:04.586544 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:04.595679 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:04.653322 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:04.676501 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:04.719620 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:04.729522 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:04.757475 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:04.776100 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:04.779134 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:04.826203 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:04.829015 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:04.868163 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:04.942354 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:08:04.987596 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:05.043368 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:05.067153 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:05.088976 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:05.205302 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:05.206048 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:05.208629 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:05.241767 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 08:08:05.390560 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:05.396037 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:05.397403 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240501 08:08:05.415729 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:05.543946 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:05.551568 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:05.582617 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:05.613082 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:05.622602 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:05.646467 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:05.686311 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:05.722950 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:05.837141 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:05.896435 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:05.948194 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:05.977371 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:06.003563 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240501 08:08:06.004127 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:06.034274 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:06.093876 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:06.269778 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:06.310381 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:06.318898 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:06.342803 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:06.385283 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:06.461961 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:06.492878 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:06.559744 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:06.575322 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:06.610517 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:06.668148 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:06.707258 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:06.759402 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:06.811090 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:06.844888 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:06.847211 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:06.902447 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:06.904857 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:06.908851 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:06.938859 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:06.961650 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:06.992202 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:07.003470 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:07.020141 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:07.049216 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:07.064672 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:07.153157 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:07.170895 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:07.183503 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:07.227156 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 08:08:07.227772 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:07.269975 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:07.276733 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:07.296838 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:07.415505 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:07.429023 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:07.433964 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:07.442400 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:07.537703 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:07.562690 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 08:08:07.658289 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:07.718339 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:07.745008 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:07.769834 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:07.791595 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:07.812249 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:08:07.830569 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:07.840828 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:07.889078 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 08:08:07.959035 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:08.010643 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:08.052888 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:08.061713 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:08.138936 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:08.138937 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:08.170295 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:08.170321 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:08.399477 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:08.453855 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:08.544574 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:08.553772 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:08.618741 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:08.695075 7250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 08:08:08.725797 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:08.749655 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:08.774494 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:08.784320 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:08.811149 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:08.848485 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:08.919896 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:08.982560 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:09.050263 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:09.055608 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:09.069605 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:09.092813 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:09.129349 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:09.210289 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:09.236244 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:09.238389 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:09.241371 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:08:09.269207 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:09.272169 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:09.272398 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:09.392853 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:09.414319 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:09.434916 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:09.452602 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:09.507790 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:09.511698 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:09.516523 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:09.596434 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:09.610632 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:09.630739 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:09.723871 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:09.757089 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:09.773047 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:09.780359 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:09.794678 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:09.808473 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:09.820405 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 08:08:09.821437 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:09.850993 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:09.857128 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:09.935950 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:10.054286 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:10.113869 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:10.121898 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:10.195219 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:08:10.263759 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:10.268131 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:10.273334 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:10.279745 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:10.302944 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:10.376865 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:10.520936 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:10.539639 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 08:08:10.574506 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:10.616617 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:10.662092 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:10.686718 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:10.806447 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:10.871647 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:10.886941 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:11.001639 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:11.010401 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:11.040458 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240501 08:08:11.139658 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:11.149686 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:11.195313 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:11.303165 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:11.304515 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:11.318310 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:11.347452 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:11.361172 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:11.432953 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:11.464619 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:11.466543 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:11.473456 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:11.518545 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:11.637310 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:11.647542 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:11.666723 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:11.674505 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:11.703166 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:11.757432 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:11.782701 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:11.845721 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:11.878801 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:11.911339 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:11.926795 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:12.006330 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:12.057264 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:12.060245 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:12.068559 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:12.157140 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:12.190153 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:12.191767 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:12.206064 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:12.207336 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 08:08:12.209415 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:12.217164 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:12.230317 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 08:08:12.230839 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:12.269368 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:12.407891 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:12.413470 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:12.444006 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:12.467463 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:12.476795 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:12.507923 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:12.529605 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:12.550731 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:12.553519 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:12.652580 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:12.763034 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:08:12.796871 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:12.817006 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:12.869189 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:12.879053 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:12.880733 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:13.000368 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:13.000437 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:13.064882 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:13.129238 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 08:08:13.252245 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:13.273363 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:13.291153 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:13.348117 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:13.415184 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:13.428236 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:13.617712 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:13.620721 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:13.674638 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:13.677318 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:13.705849 7250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 105 similar messages]
W20240501 08:08:13.706444 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:13.750739 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:13.755599 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:13.756158 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:13.768096 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:13.802709 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:13.968008 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:13.973418 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:14.005733 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:14.070413 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:14.127725 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:14.222206 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:14.234952 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:14.261055 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:14.261710 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:08:14.269146 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:14.287115 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:14.401854 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:14.480180 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:14.496304 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:14.504789 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:14.511116 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:14.523931 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:14.537357 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:14.572474 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:14.572722 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:14.693516 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:14.758157 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:14.758569 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:14.761682 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:14.780606 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:14.786181 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:14.797221 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:14.839190 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:14.889154 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 08:08:14.935309 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:14.937800 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:14.960933 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:15.047403 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:15.048696 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:15.100785 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:15.108021 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:15.174274 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:15.192348 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:15.229255 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:15.286365 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:15.328449 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:08:15.371598 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:15.513674 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:15.538313 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:15.543823 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:15.624781 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:15.677042 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:15.709043 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 08:08:15.725423 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:15.794783 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:15.804141 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:15.811432 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:16.040946 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 154 similar messages]
W20240501 08:08:16.041565 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:16.049711 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:16.057439 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:16.104436 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:16.165473 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:16.194629 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:16.246479 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:16.253971 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:16.277547 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:16.337850 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:16.343181 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:16.482424 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:16.500806 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:16.533922 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:16.542361 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:16.582933 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:16.648644 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:16.685199 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:16.738624 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:16.766530 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:16.800786 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:16.847235 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:16.854606 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:16.896816 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:16.926831 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:16.976812 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:16.994010 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:17.046635 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:17.062172 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:17.096894 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:17.158692 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:17.207500 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:17.236735 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 08:08:17.237269 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:17.253880 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:17.301294 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:17.302254 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:17.319233 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:17.356318 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:17.409934 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:17.446698 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:17.521302 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:17.562582 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:17.568866 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240501 08:08:17.582252 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:17.613869 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:17.629604 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:17.654326 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:17.723461 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:17.741142 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:17.744364 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:17.751415 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:17.790392 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:18.028839 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:18.044643 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:18.049863 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:18.070374 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:18.133455 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:18.134766 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:18.233140 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:18.260300 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:18.278789 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 08:08:18.326980 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:18.340291 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:18.404105 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:18.457283 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:18.465458 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:18.533895 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:18.538007 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:18.667248 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:18.724409 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:18.733435 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240501 08:08:18.753078 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:18.792809 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:18.809768 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:18.853677 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:18.885874 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:18.920126 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:19.009900 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:19.030210 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:19.036304 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:19.073128 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:19.080729 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:19.142771 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:19.156683 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:19.174468 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:19.250538 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:19.269096 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:19.273347 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:08:19.363068 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:19.411159 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:19.411634 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:19.443915 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:19.465565 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:19.468097 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:19.605880 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:19.663345 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:19.678061 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:19.686329 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:19.688910 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:19.709606 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:19.722038 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:19.727628 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:19.759225 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:19.770190 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:19.830977 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 08:08:19.860385 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:20.006309 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:20.070329 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:20.074234 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:20.108574 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:20.117462 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:20.130069 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:20.141794 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:20.194463 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:20.212433 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:20.222781 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:20.264492 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:20.323227 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:20.389865 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:20.521100 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:20.626467 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:20.650029 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:20.651036 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:20.659128 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:20.719173 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:20.726011 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:20.790920 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:20.795809 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:20.814519 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240501 08:08:20.834067 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:20.992163 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:21.051604 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 164 similar messages]
W20240501 08:08:21.077425 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:21.090803 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:21.116319 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:21.247642 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:21.302803 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:21.313102 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:21.313645 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:21.338179 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:21.365763 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:21.395541 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:21.460628 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:21.466667 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:21.473088 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:21.521337 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:21.630026 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:21.634418 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:21.643826 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:21.729089 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:21.769203 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:21.771987 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:21.869731 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:21.897584 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:21.903945 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:21.961835 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:21.982435 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:21.993340 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:22.041707 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:22.086326 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:22.259126 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240501 08:08:22.260524 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:22.301726 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:22.318740 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:22.332561 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:22.345527 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:22.358486 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:22.370294 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:22.408838 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 08:08:22.409554 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:22.431555 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:22.433353 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:22.450155 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:22.464913 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:22.491565 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:22.571028 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:22.729736 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:22.761054 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:22.775926 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:22.790879 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:22.892393 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:22.938881 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:22.942713 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:22.979463 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:23.006351 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:23.047598 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:23.067210 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:23.093129 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:23.132386 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:23.142781 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:23.287870 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:23.311043 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:23.393770 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:23.407676 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:23.431300 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:23.463089 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 08:08:23.498308 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:23.504695 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:23.758283 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240501 08:08:23.758991 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:23.816754 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:23.820982 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:23.880198 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:23.884794 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:23.887883 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:23.930271 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:23.934900 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:23.966702 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:24.023830 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:24.043653 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:24.080478 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:24.094669 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:24.104202 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:24.152992 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:24.278677 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:24.288501 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:08:24.291793 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:24.306466 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:24.339051 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:24.354311 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:24.538477 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:24.563097 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:24.575927 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:24.619139 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:24.621567 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:24.623560 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:24.688297 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:24.692163 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:24.725338 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:24.770859 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:24.805634 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:24.824194 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 08:08:24.862826 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:24.871263 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:24.878374 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:24.886871 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:24.931514 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:24.932166 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:24.976640 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:24.995362 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:25.087303 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:25.168388 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:25.249048 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:25.262228 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:25.292452 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:25.313198 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:25.313689 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:25.371899 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:25.372897 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:25.468286 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:25.474366 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:25.532761 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:25.599756 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:25.651160 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:25.680394 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:25.683074 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:25.701756 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:25.836124 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:25.845501 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:25.905685 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:25.983958 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 08:08:26.062330 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 160 similar messages]
W20240501 08:08:26.132602 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:26.165382 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:26.174177 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:26.209897 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:26.230159 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:26.231498 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:26.327589 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:26.352020 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:26.357378 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:26.358203 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:26.424274 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:26.446977 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:26.518096 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:26.576874 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:26.601356 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:26.639189 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:26.757170 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:26.775238 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:26.820710 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:26.825551 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20240501 08:08:26.826156 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5d18a2918c75467a9fe59979ce8936cd: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:08:26.830163 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:26.843050 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:26.863849 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:26.871856 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:26.887022 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:26.901898 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:26.966236 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:27.022038 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:27.024204 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:27.088290 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:27.108008 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:27.113641 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:27.140583 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:27.159420 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:27.233321 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:27.240053 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:27.288216 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240501 08:08:27.348335 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:27.361289 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:27.371680 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:27.436462 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:27.444074 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:27.485167 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:27.505515 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:27.542906 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:27.552851 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 08:08:27.641376 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:27.697679 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:27.757462 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:27.760627 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:27.784371 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:27.803036 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:27.902539 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:27.915001 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 21d76484f5dd499fb6fc3d534bffb779: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:08:27.971679 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:27.976857 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:28.001091 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:28.043429 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:28.068385 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:28.118566 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:28.181843 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:28.230564 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:28.237520 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:28.394549 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:28.429114 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 08:08:28.453401 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:28.608580 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:28.650240 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:28.657011 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:28.665884 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:28.671609 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:28.673357 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:28.740862 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:28.769870 7250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 08:08:28.783361 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:28.809144 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:28.858597 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:28.893016 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:28.927790 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:28.930305 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:28.996608 7160 catalog_manager.cc:4573] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cc6f7dcc930e4937865d37cb3c3bb3d2: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:08:29.055083 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:29.078410 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:29.154580 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:29.202129 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:29.255388 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:29.265868 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:29.281587 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:29.301668 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240501 08:08:29.338925 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:29.399893 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:29.414016 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:29.469576 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:29.490558 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:29.496747 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:29.508091 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:29.518364 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:29.546921 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:29.560809 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:29.620023 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:29.650933 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:29.658368 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:29.677963 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:29.685145 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:08:29.723963 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:29.760239 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:29.768285 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:29.838109 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:29.879174 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:29.936405 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 08:08:29.971019 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:29.989656 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:30.075698 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:30.127267 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:30.187233 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:08:30.197932 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:30.206683 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:30.215970 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:30.368281 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:30.494598 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:30.496057 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:30.523407 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:30.526991 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:30.544603 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:30.581684 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:30.588162 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:30.592101 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:30.636206 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:30.659504 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:30.777408 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:30.865020 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 08:08:30.871138 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:30.972548 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:30.997591 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:31.071326 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240501 08:08:31.071879 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:31.081224 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:31.081861 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:31.102406 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:31.132108 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:31.143783 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:31.264845 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:31.374212 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:31.377116 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:31.415489 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:31.455132 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:31.464924 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:31.516940 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:31.518952 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:31.556998 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:31.590729 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:31.639132 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:31.806131 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:31.936630 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:31.944716 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:31.952064 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:31.954216 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:31.956729 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:31.982690 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:32.017913 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:32.043361 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:32.081847 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:32.097481 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:32.117667 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:32.124444 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:32.141323 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:32.149873 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:32.164534 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:32.234879 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:32.279415 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:32.282975 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:08:32.365926 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 08:08:32.434520 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:32.442793 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:32.461629 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:32.461649 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 08:08:32.461680 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:32.501153 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:32.524022 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:32.532539 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:32.577488 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:32.726294 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:32.827396 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:32.949513 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:32.984643 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:32.986763 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:32.997956 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:33.004031 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:33.007932 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:33.041033 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:33.047061 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:33.083516 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 08:08:33.094444 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:33.181607 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:33.293488 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:33.344003 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:33.368903 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:33.369848 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:33.458866 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:33.468982 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:33.509753 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:33.537967 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:33.682971 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:33.718950 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:33.736120 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:33.839766 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:33.872002 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:33.926396 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 08:08:33.927008 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:33.970640 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:34.027805 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:34.032137 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:34.077224 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:34.082864 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:34.160832 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:34.161460 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:34.297672 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:34.315562 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240501 08:08:34.316251 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:34.345113 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:34.369403 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:34.376735 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:34.383373 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:34.399544 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:34.504685 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:34.519775 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:34.527091 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:34.537153 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:34.590343 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:34.620648 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:34.635185 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:34.638321 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240501 08:08:34.656891 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:34.689669 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:34.699563 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:34.735397 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:34.779695 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:34.889313 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:08:34.925251 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:34.934408 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:34.961297 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:34.979846 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:34.985038 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:35.062603 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:35.063122 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:35.160848 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:35.322772 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:35.346729 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:35.411721 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:35.498539 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:35.530845 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:35.533594 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:35.617038 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:35.672626 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:35.673753 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:35.705354 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:35.716828 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:35.737099 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:35.761426 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 08:08:35.765404 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:35.829102 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:35.839962 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:35.971702 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:36.034070 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:36.123795 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240501 08:08:36.127183 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:36.152076 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:36.160266 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:36.176268 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:36.220429 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:36.319826 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:36.326640 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:36.429121 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:36.505707 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:36.534600 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:36.565171 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:36.626009 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:36.669286 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:36.678632 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:36.693828 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:36.707525 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:36.710680 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:36.864559 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:36.875514 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:36.881688 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:36.885599 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 08:08:36.900799 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:36.926919 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:36.936874 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:36.956449 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:36.974193 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:36.998198 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:37.010399 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:37.050227 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:37.093919 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:37.118086 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:37.166824 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:37.268385 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:37.276643 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:08:37.283263 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:37.338872 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:37.342171 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:37.364053 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:37.371206 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 08:08:37.371896 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:37.441143 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:37.554801 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:37.590276 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:37.626356 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:37.630620 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:37.674932 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:37.901006 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:37.906288 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:37.940673 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:37.943751 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:38.000739 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:38.045866 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:38.092533 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:38.156971 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:38.265693 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:38.290225 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:38.338248 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240501 08:08:38.379212 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:38.433956 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:38.434736 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:38.481767 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:38.501847 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:38.505522 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:38.544771 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:38.573976 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:38.611451 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:38.611451 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:38.701411 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:38.812425 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:38.860329 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:38.879400 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:38.953955 7250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 08:08:38.991690 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:38.997913 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:39.015990 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:39.108183 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:39.109834 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:39.123122 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:39.200204 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:39.221396 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:39.251807 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:39.260216 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:39.299470 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:39.322971 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:39.325179 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:39.329856 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:08:39.373785 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 08:08:39.385098 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:39.440729 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:39.481616 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:39.501101 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:39.512689 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:39.532932 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:39.555799 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:39.560173 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:39.596652 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:39.596921 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:39.713564 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:39.726261 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:39.795433 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:08:39.867218 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:39.907811 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:39.913352 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:39.929131 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:39.995998 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:40.029596 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:40.044773 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:40.048472 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:40.060124 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:40.255192 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:40.270162 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:40.348783 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:40.389545 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:40.416760 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:40.424078 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 08:08:40.459592 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:40.546028 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:40.577172 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:40.608187 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:40.651235 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:40.682827 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:40.736588 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:40.857017 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:40.881505 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:40.938715 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:40.961002 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:40.966228 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:41.054471 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:41.069506 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:41.084039 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:41.136309 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 160 similar messages]
W20240501 08:08:41.137126 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:41.152683 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:41.171275 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:41.370854 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:41.424507 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:41.447486 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:41.480476 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:41.499984 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:41.516575 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:41.597368 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:41.652074 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:41.658033 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:41.662623 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:41.737586 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 08:08:41.747501 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:41.752835 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:41.764065 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:41.807888 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:41.824805 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:41.826699 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:41.855650 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:41.864204 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:41.898366 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:41.900127 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:41.900722 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:41.921109 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:41.923542 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:41.993527 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:42.030162 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:42.123260 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:42.185511 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:42.211010 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:42.255699 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:42.302143 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:08:42.307580 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:42.340576 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:42.401623 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 08:08:42.492290 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:42.499032 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:42.543545 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:42.578589 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:42.627347 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:42.643328 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:42.709415 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:42.746011 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:42.750285 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:42.785832 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:42.793581 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:42.812116 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:42.846515 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 08:08:42.971849 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:42.984133 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:43.138156 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:43.193598 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:43.246065 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:43.301139 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:43.312747 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:43.322667 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:43.388818 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:43.405568 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:43.409956 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:43.503228 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:43.540185 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:43.589594 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:43.676589 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:43.734326 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:43.789131 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:43.887269 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:43.937768 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:43.987008 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:44.003010 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:44.042608 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 08:08:44.043246 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:44.059731 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240501 08:08:44.082432 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:44.112198 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:44.164069 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:44.177155 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:44.185304 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:44.253648 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:44.259213 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:44.269589 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:44.288792 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:44.308552 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:44.320417 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:44.337738 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:44.337759 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:44.375124 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:44.375849 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:08:44.383385 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:44.446460 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:44.539732 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:44.580258 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:44.716552 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:44.730896 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:44.754356 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:44.763986 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:44.787285 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:44.840462 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:44.873163 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:44.890031 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:44.921751 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:44.925066 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:44.943943 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:45.009092 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:08:45.086227 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:45.164695 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:45.171026 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:45.177865 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:45.307799 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:45.337664 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:45.337795 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:45.396689 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:45.406925 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:45.424799 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:45.443966 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 08:08:45.525359 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:45.609298 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:45.656420 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:45.686549 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:45.785682 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:45.831460 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:45.880460 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:45.922444 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:46.067658 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:46.105242 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:46.147105 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:46.159974 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 159 similar messages]
W20240501 08:08:46.199555 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:46.215555 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:46.235487 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:46.242331 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:46.244129 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:46.317703 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:46.420025 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:46.431003 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:46.448746 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:46.459589 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:46.563650 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240501 08:08:46.594048 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:46.602684 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:46.621280 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:46.657578 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:46.668247 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:46.681202 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:46.700388 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:46.743188 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:46.779246 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:46.795614 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:46.817003 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:46.834056 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:46.837673 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:46.841274 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:46.874465 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:46.915066 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:47.051625 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:47.103724 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:47.110200 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:47.221609 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:47.273797 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:47.339447 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:47.379103 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:47.386615 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:47.392372 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:08:47.400935 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:47.410140 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 08:08:47.410735 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:47.440217 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:47.487107 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:47.505936 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:47.559216 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:47.595369 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:47.649670 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:47.666242 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:47.693485 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:47.694214 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:47.738401 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:47.742916 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:47.809525 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:47.874132 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:47.953581 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:47.956411 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:48.089787 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:48.160580 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240501 08:08:48.185216 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:48.246379 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:48.254715 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:48.268206 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:48.445147 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:48.491333 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:48.520149 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:48.601744 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:48.639057 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:48.689697 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:48.712450 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:48.734889 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:48.746459 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:48.807812 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:48.845428 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:48.846431 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:48.846566 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:48.867220 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:48.868073 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:48.927949 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:49.008868 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240501 08:08:49.017378 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:49.051515 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 08:08:49.052151 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:49.086344 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:49.106335 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:49.127090 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:49.183030 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:49.246137 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:49.250247 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:49.256215 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:49.275867 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:49.285391 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:49.290890 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:49.359000 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:49.383767 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:49.405223 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:49.445226 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:08:49.591923 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:49.614248 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:49.690706 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:49.718992 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:49.746703 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:49.798030 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:49.806442 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:49.876421 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:49.890578 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:49.896072 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:08:49.966296 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:50.028117 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:50.064157 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:50.069687 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:50.136379 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:50.143152 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:50.150626 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:50.161283 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:50.218022 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:50.277053 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:50.320691 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:50.353833 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:50.357843 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:50.364804 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:50.387359 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:50.509812 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240501 08:08:50.600975 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:50.617121 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:50.674562 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:50.681586 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:50.828091 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:50.961097 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:50.962289 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:51.059082 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:51.129568 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:51.166483 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240501 08:08:51.180387 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:51.251259 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:51.263866 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:51.298393 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:51.307610 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:51.310842 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:51.318329 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:51.370363 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:51.375267 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:51.402051 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:51.406975 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:51.410045 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240501 08:08:51.413002 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:51.453617 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:51.478456 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:51.543242 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:51.588639 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:51.594506 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:51.599938 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:51.693007 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:51.700145 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:51.798923 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:51.804122 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:51.835882 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:51.869660 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:51.998170 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:52.007902 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:52.086315 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:52.108999 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:52.138389 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:52.138805 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:52.143520 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:52.184104 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:52.263319 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:52.278273 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:52.312748 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:52.347095 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:52.426395 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:52.427891 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:08:52.479418 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 08:08:52.500577 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:52.532150 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:52.547850 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:52.592602 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:52.596470 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:52.617478 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:52.619030 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:52.684600 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:52.844666 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:52.901486 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:52.925763 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240501 08:08:52.983366 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:53.000273 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:53.008050 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:53.048900 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:53.063170 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:53.093946 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:53.198951 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:53.269531 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:53.330504 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:53.395318 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:53.403558 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:53.474233 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:53.553030 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:53.626818 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:53.690145 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:53.706840 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:53.752207 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:53.774513 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:53.797941 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:53.821273 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:53.835551 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:53.882644 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:53.890709 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:53.891880 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:53.920203 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:53.921259 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:53.954981 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:54.010954 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:54.013237 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240501 08:08:54.032577 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:54.164496 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:54.171416 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:54.194605 7250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240501 08:08:54.203183 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:54.238093 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:54.280993 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:54.399873 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:54.437716 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:54.451133 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:54.478672 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:08:54.514779 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:54.544135 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:54.608032 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:54.623303 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:54.625918 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:54.778151 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:54.797871 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:54.810631 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:54.824326 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:54.826690 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:54.853226 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:54.857137 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:54.890013 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:54.922654 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:54.938711 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:54.990176 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:54.992045 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:54.993320 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:55.005692 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:08:55.123553 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:55.123596 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:55.236048 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:55.271078 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:55.309664 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:55.324245 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:55.363757 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:55.393491 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:55.446561 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:55.500490 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240501 08:08:55.531899 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:55.561807 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:55.790620 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:55.810221 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:55.847316 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:55.882551 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:55.907688 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:56.035878 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:56.103699 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:56.146994 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:56.177253 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:56.178467 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240501 08:08:56.205440 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:56.205766 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:56.237236 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:56.322734 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:56.373134 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:56.389457 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:56.425036 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:56.456331 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:56.494932 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240501 08:08:56.518793 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:56.518841 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:56.563908 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:56.616160 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:56.654697 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:56.665670 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:56.681864 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:56.695228 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:56.703006 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:56.806316 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:56.816751 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:56.928382 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:56.949344 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:56.981170 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:57.034256 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:57.092398 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:57.105918 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:57.127377 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:57.136361 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:57.148780 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:57.239953 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:57.266721 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:57.312399 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:57.321138 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:57.368155 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:57.385660 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:57.410190 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:57.425383 7162 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ffc23756b44d4d588e0a8175075d9bc6 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 09a7b6e2731a474b80411efb62323604 (127.0.99.132:35789) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "09a7b6e2731a474b80411efb62323604" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35789 } } peers { permanent_uuid: "1807c0c2e7d04b20b0cda311c42eef8e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "d8aa5ac376a54e1f8417e5af11d28fa7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "1c8cb9b1059a47cc822ca13a2b931776" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41471 } attrs { promote: true } }
W20240501 08:08:57.519094 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:57.522831 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:57.554279 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:57.556326 7514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 08:08:57.567195 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:57.600632 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:57.608667 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:57.628635 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:57.699934 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:57.712401 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:57.867195 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:08:57.898859 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:57.938364 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:57.969795 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240501 08:08:58.052150 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:58.138309 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:58.152777 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:58.155321 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:58.242079 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:58.313962 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:58.321086 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:58.387871 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:58.472812 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:58.541316 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:58.570506 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:58.590991 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:58.633311 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:58.638325 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:58.648798 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:58.656491 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:58.701026 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:58.781103 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:58.831980 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:58.860407 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:58.864720 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:58.921613 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:58.945350 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240501 08:08:58.974486 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:58.995117 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:08:59.042557 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:59.181115 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:59.194510 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:59.202772 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:59.255815 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 08:08:59.266999 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:59.267917 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:59.282313 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:59.330384 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:59.331962 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:59.341146 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:59.447938 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:08:59.455459 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:59.472805 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:59.484509 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:59.569649 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:08:59.609839 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:08:59.616667 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:08:59.689759 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:59.749164 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:59.756203 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:59.793231 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:08:59.861150 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:59.892186 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:08:59.894956 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:08:59.946239 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:00.023420 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:00.032123 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:00.033289 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:00.099014 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:00.172612 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:00.186292 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:00.187400 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:00.226899 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:00.251224 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:00.332001 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:00.344393 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:00.347494 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:00.455942 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:09:00.516578 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:09:00.626453 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:00.633322 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:00.635294 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:00.635299 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:00.814771 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:00.831140 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:00.861677 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:00.862282 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240501 08:09:00.864104 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:00.962956 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:01.011786 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:01.053959 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:01.094427 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:01.102437 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:01.190842 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:01.206612 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 156 similar messages]
W20240501 08:09:01.207211 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:01.217180 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:01.228991 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:01.289005 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:01.291123 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:01.423103 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:01.424476 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:01.468420 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240501 08:09:01.471400 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:01.508810 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:01.528947 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:01.534279 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:01.599247 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:01.663815 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:01.725889 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:01.744354 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:01.749498 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:01.749759 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:01.817308 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:01.818789 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:01.845886 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:01.848927 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:01.959376 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:09:01.982774 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:02.091609 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:02.096872 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:02.118386 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:02.168004 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:02.193564 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:02.194538 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:02.291940 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:02.319878 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:02.342080 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:02.367944 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:02.377310 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:02.524639 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:02.532056 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:02.598896 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 08:09:02.604229 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:02.630837 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:02.632571 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:02.672991 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:02.687206 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:02.768160 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:02.825683 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:02.834259 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:02.853195 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:02.867866 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:02.919242 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:09:03.020153 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:03.215997 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:03.223893 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:03.227829 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:03.251155 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:03.284307 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:03.287310 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:03.345315 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:03.412540 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:03.413913 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:03.431298 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:03.449352 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:03.545924 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240501 08:09:03.587134 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:03.673238 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:03.703322 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:03.734768 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:03.753584 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:03.753670 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:03.755208 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:03.783118 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:03.812585 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:03.848524 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:03.866621 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:03.902604 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:03.943245 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:04.000274 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240501 08:09:04.078940 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:04.082931 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:04.125347 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:04.138528 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:04.150665 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:04.163100 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:04.258216 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240501 08:09:04.258850 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:04.363281 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:04.380985 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:04.385702 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:04.413089 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:04.423002 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:04.557565 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:04.566310 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:04.582651 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:04.613052 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:04.615234 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240501 08:09:04.619885 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:04.692361 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:04.713961 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:04.804045 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:04.881212 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:04.930531 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:05.012226 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:05.089888 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:05.107982 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:05.183646 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:05.205818 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:05.218434 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:05.229247 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:05.260497 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:05.295830 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:05.306923 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:05.373188 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:05.420444 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:05.442737 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:05.488147 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:05.557042 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:05.612893 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:05.615900 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:05.645188 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:09:05.691147 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:05.740216 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:05.743800 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:05.760736 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:05.767609 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:05.812629 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:05.961169 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:05.988477 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:06.011945 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:06.028352 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:06.063189 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:06.070788 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:06.095712 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240501 08:09:06.125859 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:06.142942 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:06.228209 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240501 08:09:06.244158 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:06.284308 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:06.398857 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:06.403607 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:06.423202 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:06.459578 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:06.493408 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240501 08:09:06.574590 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:06.614728 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:06.618355 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:06.665879 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:06.668244 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:06.711592 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:06.714299 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:06.792940 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:06.821691 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:06.827109 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:06.866159 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:06.935834 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:06.957954 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:07.033082 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:07.118901 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:07.127040 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:07.130300 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:07.183136 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:07.207146 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:07.284194 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:07.317677 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:07.457470 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:07.513916 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:07.527360 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:07.558907 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:07.559626 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:07.580789 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:07.591938 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:07.611793 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:07.628356 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 08:09:07.748162 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:07.748874 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:07.781311 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:07.806465 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:07.819952 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:07.905854 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:07.919847 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:08.048995 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:08.081812 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:08.116544 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:09:08.162176 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:08.163442 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:08.182747 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:08.195569 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:08.222851 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:08.235287 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:08.266183 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240501 08:09:08.286615 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:08.307318 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:08.320978 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:08.387857 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:08.485457 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:08.526988 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:08.536652 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:08.583408 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:08.590474 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:08.679445 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:08.715960 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:08.732386 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:08.847349 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:09.066499 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240501 08:09:09.082782 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:09.111516 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:09.114338 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:09.125298 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:09.142931 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:09.146314 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:09.161761 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:09.172070 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:09.174217 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:09.217028 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:09.222802 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:09.225904 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:09.259560 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:09.285017 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240501 08:09:09.285734 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:09.353804 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:09.438827 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:09.455121 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:09.536893 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:09.622442 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:09.627066 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240501 08:09:09.639397 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:09.690407 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:09.734584 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:09.763705 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:09.783918 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:09.823575 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:09.906328 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:09.950740 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:10.015326 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:10.043763 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:10.107220 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:10.122119 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:10.155900 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:10.196311 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:10.242197 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:10.246491 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:10.364064 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:10.386708 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:10.422222 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:10.474493 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:10.514278 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240501 08:09:10.531704 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:10.657300 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:10.690497 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:10.698678 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:10.710526 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:10.758683 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:10.759477 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:10.770577 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:10.814246 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:09:10.842155 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:10.865437 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:10.884083 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:10.904597 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:10.917094 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:10.993803 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:11.097267 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:11.126178 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:11.153036 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:11.233317 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 163 similar messages]
W20240501 08:09:11.233839 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:11.237128 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:11.277210 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:11.492751 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:11.521687 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:11.527149 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:11.535511 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:11.550772 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:11.557274 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:11.642246 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:11.652760 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:11.776711 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240501 08:09:11.780807 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:11.789462 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:11.809720 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:11.827776 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:11.831231 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:11.838450 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:11.858769 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:11.941964 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:11.971239 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:12.030894 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:12.050458 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:12.158066 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:12.162583 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:12.184576 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:12.215452 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:12.226178 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:12.289820 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:12.380154 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:12.431268 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:12.450970 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:12.458058 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:12.502038 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:12.522492 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:12.578840 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:12.597018 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:12.605605 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:12.630838 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 08:09:12.640331 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:12.646008 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:12.735760 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:12.867283 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:12.898890 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:12.910804 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:13.001040 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:13.070960 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:13.107367 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:13.121650 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240501 08:09:13.141238 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:13.160203 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:13.199829 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:13.279592 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:13.381937 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:13.393425 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:13.411124 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:13.481472 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:13.486788 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:13.519842 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:13.528244 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:13.546265 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:13.588033 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:13.616295 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:13.619189 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:13.629415 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:13.637687 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:13.769809 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:13.783033 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:09:13.795567 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:13.802448 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:13.942142 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:14.031301 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:14.083290 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:14.171332 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:14.174424 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:14.187623 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:14.222448 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240501 08:09:14.286551 7250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 08:09:14.287103 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:14.312877 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:14.335181 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:14.396775 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:14.401122 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:14.407348 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:14.448316 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:14.520041 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:14.547103 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:14.571599 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:14.617350 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:14.632776 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240501 08:09:14.718134 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:14.726055 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:14.729605 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:14.742434 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:14.749747 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:14.762347 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:14.787387 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:14.808743 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:14.833636 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:14.907183 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:14.939733 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:14.987427 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:14.996188 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:15.062682 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:15.076463 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:15.089077 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:15.163022 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:15.232331 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:15.260691 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:15.269094 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:15.500560 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:15.501447 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:15.533304 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240501 08:09:15.557758 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:15.615046 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:15.639914 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:15.758198 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:15.825594 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:15.864908 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:15.864881 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:15.879552 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:15.899153 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:15.934543 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:15.993024 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:16.003161 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:16.016626 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:16.056733 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:16.058614 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:16.164299 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:09:16.206247 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:16.239851 7644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 160 similar messages]
W20240501 08:09:16.270298 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:16.305121 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:16.395013 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:16.433559 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:16.522346 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:16.536201 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:16.538271 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240501 08:09:16.541095 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:16.578015 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:16.578583 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:16.664172 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:16.698643 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:16.700601 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:16.778924 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:16.783017 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:16.885895 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:16.926885 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:16.930984 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:16.971997 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:16.977128 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:17.000578 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:17.119908 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:17.132416 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:17.138543 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:17.188802 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:17.244431 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:17.254707 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:17.295375 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:17.304817 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:17.314914 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:17.349685 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:17.365463 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:17.477841 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:17.489821 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:17.515561 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:17.622138 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:17.632956 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 08:09:17.655357 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:17.687567 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:17.710590 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:17.718457 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:17.752713 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:17.767419 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:17.822237 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:17.917520 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:17.945943 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:17.950783 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:18.119004 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:18.131196 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240501 08:09:18.229665 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:18.341974 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:18.353511 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:09:18.382370 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:18.401520 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:18.406070 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:18.433753 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:18.434476 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:18.529884 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:18.562896 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:18.623134 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:18.642869 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:18.672629 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:18.684149 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:18.684250 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:18.713930 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:18.727692 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:18.856252 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:18.924302 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:18.944928 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:18.946285 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:18.972599 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240501 08:09:19.037050 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:19.070900 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:19.084231 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:19.172641 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:19.177645 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:19.216118 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:19.229914 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:19.247145 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:19.307235 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 08:09:19.308506 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:19.343212 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:19.418802 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:19.419845 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:19.437965 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:19.443533 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:19.444324 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:19.451398 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:19.469700 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:19.627499 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:19.631258 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:19.652797 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:09:19.653488 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:19.659097 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:19.694531 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:19.711036 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:19.885720 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:19.887428 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:19.986523 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:19.999322 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:20.042380 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:20.047099 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:20.082042 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:20.156268 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240501 08:09:20.237532 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:20.271543 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:20.328456 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:20.334403 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:20.386461 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:20.410043 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:20.417331 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:20.425911 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:20.525101 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:20.529732 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:09:20.545447 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:20.676564 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:20.812031 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:20.829473 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:20.845019 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:20.846501 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:20.864050 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:20.912830 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:20.965651 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:21.056931 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:21.102154 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:21.131834 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:21.165854 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:21.197319 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:21.237335 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:21.258952 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240501 08:09:21.259538 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:21.261654 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:21.337742 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:21.364684 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:21.379112 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:21.457890 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:21.518795 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:21.589780 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:21.592200 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240501 08:09:21.599318 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:21.601058 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:21.618904 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:21.666426 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:21.725049 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:21.737088 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:21.759902 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:21.766569 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:21.794332 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:21.800868 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:21.891378 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:21.981968 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:22.064628 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:22.136574 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:22.175253 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:22.232295 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:22.234292 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:22.256532 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:22.279397 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:22.397922 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:22.405736 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:22.413136 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:22.508420 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:22.562824 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:22.590173 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:22.640816 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 08:09:22.702081 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:22.749100 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:22.759500 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:22.775759 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:22.794391 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:22.795799 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240501 08:09:22.845402 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:22.866288 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:22.926208 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:22.952461 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:22.999188 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:23.008709 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:23.024962 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:23.190552 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:23.253530 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:23.320775 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:23.337121 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:23.342499 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:23.378998 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:23.414497 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:23.415409 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:09:23.430316 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:23.629490 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:23.634876 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:23.646590 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:23.693176 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:23.714251 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:23.720012 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:23.721295 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:23.845153 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:23.847651 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:23.911124 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:24.003952 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:24.023087 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:24.031138 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:24.033648 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:24.070024 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:24.072275 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240501 08:09:24.072820 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:24.074891 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:24.078661 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:24.139381 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:24.193859 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:24.274580 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:24.278049 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:24.299485 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:24.346453 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240501 08:09:24.347167 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:24.370446 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:24.375716 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:24.462093 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:24.476794 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:24.552122 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:24.624203 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:24.728541 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:24.770432 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:24.795264 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:24.805449 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:24.893630 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:09:24.902340 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:24.911268 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:24.918092 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:24.930323 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:24.999329 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:25.005844 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:25.038837 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:25.100366 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:25.106799 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:25.207754 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:25.356663 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:25.392959 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:25.393276 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:25.414494 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:25.429711 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240501 08:09:25.437176 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:25.471717 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:25.507779 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:25.597241 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:25.601286 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:25.725958 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:25.783430 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:25.797945 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:25.828485 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:25.898300 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:25.995479 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:25.995828 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:25.996973 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:26.019307 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:26.028395 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:26.071640 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:26.172400 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:26.185148 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:09:26.302879 7645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240501 08:09:26.303514 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:26.359858 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:26.360514 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:26.377725 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:26.424800 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:26.428637 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:26.431174 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:26.436278 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240501 08:09:26.534312 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:26.550383 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:26.560891 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:26.640990 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:26.731417 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:26.747289 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:26.760511 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:26.762305 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:26.767493 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:26.775297 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:26.836347 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:26.845211 7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20240501 08:09:26.845903 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5d18a2918c75467a9fe59979ce8936cd: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:09:26.861438 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:26.914449 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:26.938062 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:27.010982 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:27.018708 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:27.055162 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:27.160431 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:27.265224 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:27.265275 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:27.311893 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:27.394279 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:27.407892 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:27.451730 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:27.452699 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:27.522485 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:27.555583 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:27.566493 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:27.597206 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:27.598291 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:27.666599 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 08:09:27.713667 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:27.756475 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:27.820269 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:27.838599 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:27.916374 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:27.918740 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:27.927119 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 21d76484f5dd499fb6fc3d534bffb779: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:09:27.976569 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:28.044909 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:28.048631 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:28.074050 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:28.097847 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:28.102566 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240501 08:09:28.129647 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:28.187400 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:28.206357 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:28.231050 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:28.246928 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:28.288254 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:28.455777 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:28.474973 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:09:28.550788 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:28.555034 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:28.639587 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:28.667716 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:28.701843 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:28.704214 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:28.721449 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:28.728405 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:28.862419 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:28.866427 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:28.942449 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:28.960894 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:29.003868 7160 catalog_manager.cc:4573] TS 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cc6f7dcc930e4937865d37cb3c3bb3d2: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111)
W20240501 08:09:29.035341 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:29.036625 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:29.116539 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:29.127547 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240501 08:09:29.175184 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:29.194247 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:29.219331 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:29.227450 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:29.284255 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:29.399436 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240501 08:09:29.400964 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:29.432402 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:29.447870 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:29.461493 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:29.475472 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:29.494028 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:29.495226 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:29.497246 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:29.503975 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:29.685479 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:29.696059 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:29.700497 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:29.709467 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:29.834843 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:29.838745 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:29.848291 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:29.859267 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:29.885615 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:29.898548 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:29.919786 7381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:09:30.055853 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:30.077553 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:30.118662 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:30.151404 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:30.182808 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:30.215838 7160 catalog_manager.cc:4573] TS 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b8fdc2b9c2ef44b0bfec59a1a2f6b626: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111)
W20240501 08:09:30.257488 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:30.312829 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:30.344993 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:30.359421 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:30.371436 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:30.373682 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:30.487603 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:30.588769 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:30.600979 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:30.638959 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:30.672626 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:30.726294 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:30.770172 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:30.776744 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:30.784749 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:30.818368 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:30.853869 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:30.885794 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240501 08:09:30.901556 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:09:30.988664 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:31.095639 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:31.149682 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:31.193106 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:31.265429 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:31.278581 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:31.288827 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:31.296974 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:31.303525 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:31.323769 7643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 163 similar messages]
W20240501 08:09:31.362393 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240501 08:09:31.370869 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:31.389436 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:31.404472 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:31.471580 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:31.569037 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:31.592545 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:31.604849 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:31.626165 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:31.658562 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:31.696254 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:31.710723 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:31.737391 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:31.804733 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:31.882707 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:31.906942 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:31.981643 7252 consensus_peers.cc:485] T e178f7c3c17c4e1ca3d16b6441f76e77 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:32.016741 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:32.037441 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:32.051159 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:32.069363 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:32.074834 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:32.100330 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:32.108891 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:32.174142 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:32.249420 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:32.362607 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:32.385354 7383 consensus_peers.cc:485] T 8882528b26974c7aa73bb0ff0c444836 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:32.403781 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:32.428498 7645 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:32.467329 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:32.512421 7512 consensus_peers.cc:485] T b8fdc2b9c2ef44b0bfec59a1a2f6b626 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:32.549083 7645 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:32.583528 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:32.616185 7512 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:32.662586 7252 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:32.697847 7512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 08:09:32.734184 7250 consensus_peers.cc:485] T 0e42ef8df4884bc9bc33f5ed4c164c8c P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:32.828475 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:32.850248 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:32.866744 7514 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:32.962092 7512 consensus_peers.cc:485] T cc6f7dcc930e4937865d37cb3c3bb3d2 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:32.968178 7644 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:33.008283 7643 consensus_peers.cc:485] T 4dde9710a5eb4baa881d9dd04affe184 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:33.079545 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:33.095116 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:33.100598 7381 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:33.140980 7512 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:33.181879 7381 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:33.259387 7644 consensus_peers.cc:485] T 1e44704a9b9c46cf954d8e6937983841 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240501 08:09:33.270186 7250 consensus_peers.cc:485] T 1088a116ccec4020aec562a69791a188 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:33.290608 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:33.380707 7643 consensus_peers.cc:485] T 28baea6d7d3f442ba1dbfc5b37de90ac P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:09:33.438844 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:33.503203 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:33.511600 7251 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:33.525271 7381 consensus_peers.cc:485] T dd17760cd5c2454db97610b2f71c0ee2 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:33.656579 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:33.675779 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:33.763147 7645 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:33.799810 7252 consensus_peers.cc:485] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:33.810263 7514 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:33.824807 7513 consensus_peers.cc:485] T cd07a00b6817400f86bf8c32bd592d8b P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:33.875178 7383 consensus_peers.cc:485] T 3abb5202d8d24d9b9a67f7a87b7300a6 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:33.928280 7513 consensus_peers.cc:485] T 7377810e08104526b8b8acdf886a3368 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240501 08:09:33.942977 7252 consensus_peers.cc:485] T 02b42b5fd3d440ea950b35ed461b8e12 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:33.945164 7645 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:09:33.964162 7513 consensus_peers.cc:485] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:33.973054 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:33.983029 7643 consensus_peers.cc:485] T df6dd3d8482d4931b1d3014a67e739c6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:34.040505 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:34.042992 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:34.119577 7512 consensus_peers.cc:485] T bde42e485c4b49ce8acd694a25ba683a P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:34.188119 7643 consensus_peers.cc:485] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:34.227445 7644 consensus_peers.cc:485] T 4ea0db42a75a4879b3d3d608666dc2bf P 09a7b6e2731a474b80411efb62323604 -> Peer 3b69c5d799e14e438df760453438aa75 (127.0.99.134:33589): Couldn't send request to peer 3b69c5d799e14e438df760453438aa75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:33589: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:34.256683 7645 consensus_peers.cc:462] T 81ccdd855b584d65860fd7a5777920a3 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:34.306536 7250 consensus_peers.cc:485] T 5970772c02d04e0b95a2f2feb7dcd655 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:34.313854 7512 consensus_peers.cc:485] T 21d76484f5dd499fb6fc3d534bffb779 P 31b02f06be8a4455b94b1254d77005bd -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:34.351011 7382 consensus_peers.cc:485] T f647cfc562794e65a05918b241014d72 P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:09:34.462807 7644 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1807c0c2e7d04b20b0cda311c42eef8e (127.0.99.136:42587): Couldn't send request to peer 1807c0c2e7d04b20b0cda311c42eef8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:09:34.478631 7250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 08:09:34.479259 7250 consensus_peers.cc:462] T 3d8ae3c82c9d4c5c858fa3f9d5959eb7 P b4f8020266b5455b872a2973fc396578 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:34.486245 7643 consensus_peers.cc:462] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111)
W20240501 08:09:34.560806 7645 consensus_peers.cc:485] T 5a37e631ae4f46a2be1ad5c43db5724b P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:09:34.570111 7381 consensus_peers.cc:485] T 5d18a2918c75467a9fe59979ce8936cd P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer 1c8cb9b1059a47cc822ca13a2b931776 (127.0.99.135:41471): Couldn't send request to peer 1c8cb9b1059a47cc822ca13a2b931776. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41471: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:09:34.593313 7514 consensus_peers.cc:485] T 8311d0921e9e417090e441884615525a P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:34.635311 7514 consensus_peers.cc:462] T 4297f95243ac431099a8279c3d9e4b25 P 31b02f06be8a4455b94b1254d77005bd -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111)
W20240501 08:09:34.672056 7383 consensus_peers.cc:485] T 54ced80b9ff943ec88c27b7842500b3d P d6b4be7bbd1649d0b7c5e8efe81f759f -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:34.722494 7645 consensus_peers.cc:485] T ffc23756b44d4d588e0a8175075d9bc6 P 09a7b6e2731a474b80411efb62323604 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn't send request to peer d8aa5ac376a54e1f8417e5af11d28fa7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33901: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:09:34.766572 7252 consensus_peers.cc:485] T b357c740aca144b7b063845726f419e2 P b4f8020266b5455b872a2973fc396578 -> Peer d8aa5ac376a54e1f8417e5af11d28fa7 (127.0.99.133:33901): Couldn'